CIHealth
<
Weekly: Oct 26 - Nov 1▾
Last 7 Days Weekly: Oct 26 - Nov 1 Sprint: Oct 26 - Nov 8 Single Day: Nov 1
Custom time window
> Reset
View JSON API

Environment: PROD

Failure patterns: 42 · Failures matched: 153 · Total runs: 51 · Runs affected: 11

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...
full failure pattern:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
Signal: NoiseSignal: Noise — struct/object fragment
e2e11.96%1 of 51 job runs affectedNoiseSignal: Noise — struct/object fragmentShow trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
    <*fmt.wrapError | 0x27a8bf192000>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x27a8c0a02f80>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
    <*fmt.wrapError | 0x27a8bf192000>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x27a8c0a02f80>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2114854607891992576
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e47.84%4 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 19 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0DEV, INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-s7k2sd" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x297062e2e820>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-s7k2sd: failed waiting for cluster="cp-ystream-upgrade-4-23-s7k2sd" in resourcegroup="rg-cp-ystream-upgrade-4-23-np7xjvwc8xct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460",
      "name": "5d9c960a-8f4b-476d-bd5b-132c0ce76460",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:32.825742556Z",
      "endTime": "2026-10-28T20:41:35.067951058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
            correlation ID: 00646159-dae9-4d36-b0f6-38a33eb4750d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-s7k2sd" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x297062e2e820>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-s7k2sd: failed waiting for cluster="cp-ystream-upgrade-4-23-s7k2sd" in resourcegroup="rg-cp-ystream-upgrade-4-23-np7xjvwc8xct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460",
      "name": "5d9c960a-8f4b-476d-bd5b-132c0ce76460",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:32.825742556Z",
      "endTime": "2026-10-28T20:41:35.067951058Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
            correlation ID: 00646159-dae9-4d36-b0f6-38a33eb4750d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x10164f31ca00>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p669mb2thnqj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b",
      "name": "f26dc7e7-f380-4b37-87cb-db720fa2be9b",
      "status": "Failed",
      "startTime": "2026-10-28T19:27:58.648367476Z",
      "endTime": "2026-10-28T19:57:03.816857754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
            correlation ID: b3870728-f2f7-48de-a0fa-7f5442586990
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x10164f31ca00>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p669mb2thnqj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b",
      "name": "f26dc7e7-f380-4b37-87cb-db720fa2be9b",
      "status": "Failed",
      "startTime": "2026-10-28T19:27:58.648367476Z",
      "endTime": "2026-10-28T19:57:03.816857754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
            correlation ID: b3870728-f2f7-48de-a0fa-7f5442586990
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-lrsdfl" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3c350f225e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-lrsdfl: failed waiting for cluster="cp-ystream-upgrade-4-21-lrsdfl" in resourcegroup="rg-cp-ystream-upgrade-4-21-vfbxfhgg5xwh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616",
      "name": "034eed87-a977-47cf-8ff5-2b8292aa4616",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:34.208698264Z",
      "endTime": "2026-10-28T20:41:35.072616907Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
            correlation ID: 880b9f59-ff57-4ef3-9ec1-2561f0663b62
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-lrsdfl" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x3c350f225e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-lrsdfl: failed waiting for cluster="cp-ystream-upgrade-4-21-lrsdfl" in resourcegroup="rg-cp-ystream-upgrade-4-21-vfbxfhgg5xwh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616",
      "name": "034eed87-a977-47cf-8ff5-2b8292aa4616",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:34.208698264Z",
      "endTime": "2026-10-28T20:41:35.072616907Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
            correlation ID: 880b9f59-ff57-4ef3-9ec1-2561f0663b62
            client request ID: unavailable
occurred
Contributing tests (16)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.213
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
n/a2115132439494922240
n/a2115079131212288000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 16 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x22868c3a2660>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
      "name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:09.588021132Z",
      "endTime": "2026-10-28T19:53:13.758028769Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
            correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x22868c3a2660>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
      "name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:09.588021132Z",
      "endTime": "2026-10-28T19:53:13.758028769Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
            correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x263f7a1ecb40>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
      "name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:34.589492538Z",
      "endTime": "2026-10-28T20:43:34.88292258Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
            correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x263f7a1ecb40>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
      "name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:34.589492538Z",
      "endTime": "2026-10-28T20:43:34.88292258Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
            correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x2d29037be7e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
      "name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:38.563887566Z",
      "endTime": "2026-10-28T20:41:44.890635353Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
            correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x2d29037be7e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
      "name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:38.563887566Z",
      "endTime": "2026-10-28T20:41:44.890635353Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
            correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
            client request ID: unavailable
occurred
Contributing tests (13)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 14 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
    <*fmt.wrapError | 0x15a8256fc4a0>: 
    failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
      "name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
      "status": "Failed",
      "startTime": "2026-10-28T20:04:15.708912895Z",
      "endTime": "2026-10-28T20:33:16.216619801Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
            correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
    <*fmt.wrapError | 0x15a8256fc4a0>: 
    failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
      "name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
      "status": "Failed",
      "startTime": "2026-10-28T20:04:15.708912895Z",
      "endTime": "2026-10-28T20:33:16.216619801Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
            correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xd149dec0ae0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
      "name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:20.936404311Z",
      "endTime": "2026-10-28T19:53:23.678300779Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
            correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xd149dec0ae0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
      "name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:20.936404311Z",
      "endTime": "2026-10-28T19:53:23.678300779Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
            correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x3b3d00e688d0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
      "name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
      "status": "Failed",
      "startTime": "2026-10-28T20:59:31.361013478Z",
      "endTime": "2026-10-28T21:28:40.621642045Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
            correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x3b3d00e688d0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
      "name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
      "status": "Failed",
      "startTime": "2026-10-28T20:59:31.361013478Z",
      "endTime": "2026-10-28T21:28:40.621642045Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
            correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
            client request ID: unavailable
occurred
Contributing tests (11)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.12
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 12 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
    <*fmt.wrapError | 0xba6b52aa80>: 
    failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
      "name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
      "status": "Failed",
      "startTime": "2026-10-28T20:10:20.02531103Z",
      "endTime": "2026-10-28T20:39:25.022579343Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
            correlation ID: 15324628-570b-42b9-80a7-dbd435939211
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
    <*fmt.wrapError | 0xba6b52aa80>: 
    failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
      "name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
      "status": "Failed",
      "startTime": "2026-10-28T20:10:20.02531103Z",
      "endTime": "2026-10-28T20:39:25.022579343Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
            correlation ID: 15324628-570b-42b9-80a7-dbd435939211
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
    <*fmt.wrapError | 0x104ef3fa7e0>: 
    failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
      "name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
      "status": "Failed",
      "startTime": "2026-10-28T20:10:48.049950576Z",
      "endTime": "2026-10-28T20:39:55.034999436Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
            correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
    <*fmt.wrapError | 0x104ef3fa7e0>: 
    failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
      "name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
      "status": "Failed",
      "startTime": "2026-10-28T20:10:48.049950576Z",
      "endTime": "2026-10-28T20:39:55.034999436Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
            correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x1da2d78427a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
      "name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
      "status": "Failed",
      "startTime": "2026-10-28T19:26:05.081696809Z",
      "endTime": "2026-10-28T19:55:13.864538233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
            correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x1da2d78427a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
      "name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
      "status": "Failed",
      "startTime": "2026-10-28T19:26:05.081696809Z",
      "endTime": "2026-10-28T19:55:13.864538233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
            correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
            client request ID: unavailable
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
full failure pattern:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 25: 0 · Oct 26: 5 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
    <*errors.errorString | 0x3306da82b390>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
    <*errors.errorString | 0x3306da82b390>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xdf906826d40>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0xdf906826d40>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x3454a17e2bc0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x3454a17e2bc0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
Contributing tests (8)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2114899906912063488
n/a2114854607891992576
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x34cbd3d68900>: 
    failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
      "name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:32.731497539Z",
      "endTime": "2026-10-28T20:43:34.905565303Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
            correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x34cbd3d68900>: 
    failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
      "name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:32.731497539Z",
      "endTime": "2026-10-28T20:43:34.905565303Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
            correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x118c82e68960>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
      "name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
      "status": "Failed",
      "startTime": "2026-10-28T20:58:03.268943167Z",
      "endTime": "2026-10-28T21:27:10.630246232Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
            correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x118c82e68960>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
      "name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
      "status": "Failed",
      "startTime": "2026-10-28T20:58:03.268943167Z",
      "endTime": "2026-10-28T21:27:10.630246232Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
            correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
    <*fmt.wrapError | 0x3e94b832e820>: 
    failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
      "name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
      "status": "Failed",
      "startTime": "2026-10-28T20:06:06.856819198Z",
      "endTime": "2026-10-28T20:35:15.000751357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
            correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
    <*fmt.wrapError | 0x3e94b832e820>: 
    failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
      "name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
      "status": "Failed",
      "startTime": "2026-10-28T20:06:06.856819198Z",
      "endTime": "2026-10-28T20:35:15.000751357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
            correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
            client request ID: unavailable
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x1d67385a4720>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
      "name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:18.787179838Z",
      "endTime": "2026-10-28T19:53:23.770490111Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
            correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x1d67385a4720>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
      "name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:18.787179838Z",
      "endTime": "2026-10-28T19:53:23.770490111Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
            correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x23a60cd6e680>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
      "name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:25.094565507Z",
      "endTime": "2026-10-28T19:53:33.692195544Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
            correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x23a60cd6e680>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
      "name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:25.094565507Z",
      "endTime": "2026-10-28T19:53:33.692195544Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
            correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x34142edd6a40>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
      "name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:14.723008935Z",
      "endTime": "2026-10-28T19:53:23.809094912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
            correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x34142edd6a40>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
      "name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:14.723008935Z",
      "endTime": "2026-10-28T19:53:23.809094912Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
            correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
            client request ID: unavailable
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x2a9a9c302780>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
      "name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:46.976794807Z",
      "endTime": "2026-10-28T20:43:55.053369978Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
            correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x2a9a9c302780>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
      "name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:46.976794807Z",
      "endTime": "2026-10-28T20:43:55.053369978Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
            correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3f72bfdcc60>: 
    failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
      "name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
      "status": "Failed",
      "startTime": "2026-10-28T14:39:56.97734875Z",
      "endTime": "2026-10-28T15:08:58.578863234Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
            correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3f72bfdcc60>: 
    failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
      "name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
      "status": "Failed",
      "startTime": "2026-10-28T14:39:56.97734875Z",
      "endTime": "2026-10-28T15:08:58.578863234Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
            correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x32cee6f94b40>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
      "name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
      "status": "Failed",
      "startTime": "2026-10-28T14:39:19.663998842Z",
      "endTime": "2026-10-28T15:08:28.451578554Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
            correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x32cee6f94b40>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
      "name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
      "status": "Failed",
      "startTime": "2026-10-28T14:39:19.663998842Z",
      "endTime": "2026-10-28T15:08:28.451578554Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
            correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
            client request ID: unavailable
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xe23c97da540>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
      "name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
      "status": "Failed",
      "startTime": "2026-10-28T20:28:51.316103778Z",
      "endTime": "2026-10-28T20:57:55.004654509Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
            correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xe23c97da540>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
      "name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
      "status": "Failed",
      "startTime": "2026-10-28T20:28:51.316103778Z",
      "endTime": "2026-10-28T20:57:55.004654509Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
            correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x38a796b1a740>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
      "name": "499291aa-b279-429d-80ba-7786fc6cfa93",
      "status": "Failed",
      "startTime": "2026-10-28T20:38:17.873619905Z",
      "endTime": "2026-10-28T21:07:24.99192406Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
            correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x38a796b1a740>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
      "name": "499291aa-b279-429d-80ba-7786fc6cfa93",
      "status": "Failed",
      "startTime": "2026-10-28T20:38:17.873619905Z",
      "endTime": "2026-10-28T21:07:24.99192406Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
            correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x1c7b6c0a8440>: 
    failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
      "name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
      "status": "Failed",
      "startTime": "2026-10-28T20:56:27.61784978Z",
      "endTime": "2026-10-28T21:25:30.606619249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
            correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x1c7b6c0a8440>: 
    failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
      "name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
      "status": "Failed",
      "startTime": "2026-10-28T20:56:27.61784978Z",
      "endTime": "2026-10-28T21:25:30.606619249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
            correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
            client request ID: unavailable
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
full failure pattern:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 25: 0 · Oct 26: 4 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
    <*fmt.wrapError | 0x625ce9962c0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x625d05dc8e0>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
    <*fmt.wrapError | 0x625ce9962c0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x625d05dc8e0>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
    <*fmt.wrapError | 0xb6fec4b44e0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
            observation: <*errors.errorString | 0xb6fee7deaa0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
    <*fmt.wrapError | 0xb6fec4b44e0>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
            observation: <*errors.errorString | 0xb6fee7deaa0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
    <*fmt.wrapError | 0x8108240e540>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x8108240e520>{
            observation: <*errors.errorString | 0x81084000ba0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x8108240e500>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
    <*fmt.wrapError | 0x8108240e540>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*verifiers.pollObservationError | 0x8108240e520>{
            observation: <*errors.errorString | 0x81084000ba0>{
                s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
            },
            interruption: <*fmt.wrapError | 0x8108240e500>{
                msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2114899906912063488
n/a2114854607891992576
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x2c03b21d6c20>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
      "name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:37.658468995Z",
      "endTime": "2026-10-28T20:43:44.89685741Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
            correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x2c03b21d6c20>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
      "name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:37.658468995Z",
      "endTime": "2026-10-28T20:43:44.89685741Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
            correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0xbe0fc68500>: 
    failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
      "name": "9e35dee8-3994-4ba7-9234-74a03d742751",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:15.512939467Z",
      "endTime": "2026-10-28T15:50:17.639783703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
            correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0xbe0fc68500>: 
    failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
      "name": "9e35dee8-3994-4ba7-9234-74a03d742751",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:15.512939467Z",
      "endTime": "2026-10-28T15:50:17.639783703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
            correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x24f1c7fcefa0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
      "name": "3ee91092-37d4-4687-8899-086b94094407",
      "status": "Failed",
      "startTime": "2026-10-28T13:42:47.010012275Z",
      "endTime": "2026-10-28T14:11:50.916418144Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
            correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x24f1c7fcefa0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
      "name": "3ee91092-37d4-4687-8899-086b94094407",
      "status": "Failed",
      "startTime": "2026-10-28T13:42:47.010012275Z",
      "endTime": "2026-10-28T14:11:50.916418144Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
            correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
            client request ID: unavailable
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x14a5e2008300>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
      "name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
      "status": "Failed",
      "startTime": "2026-10-28T19:26:12.743400806Z",
      "endTime": "2026-10-28T19:55:13.7057976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
            correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x14a5e2008300>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
      "name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
      "status": "Failed",
      "startTime": "2026-10-28T19:26:12.743400806Z",
      "endTime": "2026-10-28T19:55:13.7057976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
            correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x1683f5097100>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
      "name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
      "status": "Failed",
      "startTime": "2026-10-28T19:28:18.023653922Z",
      "endTime": "2026-10-28T19:57:23.841118908Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
            correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x1683f5097100>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
      "name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
      "status": "Failed",
      "startTime": "2026-10-28T19:28:18.023653922Z",
      "endTime": "2026-10-28T19:57:23.841118908Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
            correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x13016dc5e5e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
      "name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:34.851456324Z",
      "endTime": "2026-10-28T20:41:34.873934441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
            correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x13016dc5e5e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
      "name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
      "status": "Failed",
      "startTime": "2026-10-28T20:12:34.851456324Z",
      "endTime": "2026-10-28T20:41:34.873934441Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
            correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
            client request ID: unavailable
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xfa629f24ea0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xfa629c70d20>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xfa629f24ea0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xfa629c70d20>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapErrors | 0x1f0baa544120>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1f0baa40b610>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
            },
            <*fmt.wrapErrors | 0x1f0baa5440c0>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapErrors | 0x1f0baa544120>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1f0baa40b610>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
            },
            <*fmt.wrapErrors | 0x1f0baa5440c0>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x22e10818450>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x22e1036b8e0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x22e10818450>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x22e1036b8e0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115519015970410496
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x341fcf187480>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
      "name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
      "status": "Failed",
      "startTime": "2026-10-27T21:18:08.702878641Z",
      "endTime": "2026-10-27T21:37:14.197553955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
            correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x341fcf187480>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
      "name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
      "status": "Failed",
      "startTime": "2026-10-27T21:18:08.702878641Z",
      "endTime": "2026-10-27T21:37:14.197553955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
            correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2462c6c0b540>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-bl9lknth6jps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
      "name": "97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
      "status": "Failed",
      "startTime": "2026-10-27T18:48:30.242794404Z",
      "endTime": "2026-10-27T19:07:38.465503369Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
            correlation ID: f85455a5-fb5a-4cdd-8e2d-d003c7fb917e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2462c6c0b540>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-bl9lknth6jps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
      "name": "97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
      "status": "Failed",
      "startTime": "2026-10-27T18:48:30.242794404Z",
      "endTime": "2026-10-27T19:07:38.465503369Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
            correlation ID: f85455a5-fb5a-4cdd-8e2d-d003c7fb917e
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115176579041595392
n/a2115132439494922240
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e23.92%2 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x12961691c220>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
      "name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
      "status": "Failed",
      "startTime": "2026-10-28T02:05:53.148166937Z",
      "endTime": "2026-10-28T02:34:59.674611282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
            correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x12961691c220>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
      "name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
      "status": "Failed",
      "startTime": "2026-10-28T02:05:53.148166937Z",
      "endTime": "2026-10-28T02:34:59.674611282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
            correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1138be74e660>: 
    failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
      "name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
      "status": "Failed",
      "startTime": "2026-10-26T17:26:50.386134218Z",
      "endTime": "2026-10-26T17:55:54.373104457Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
            correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1138be74e660>: 
    failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
      "name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
      "status": "Failed",
      "startTime": "2026-10-26T17:26:50.386134218Z",
      "endTime": "2026-10-26T17:55:54.373104457Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
            correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
            client request ID: unavailable
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2115262480262369280
n/a2114749140742180864
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc983e72b20>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
      "name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:20.13733405Z",
      "endTime": "2026-10-28T19:53:23.880336461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
            correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc983e72b20>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
      "name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:20.13733405Z",
      "endTime": "2026-10-28T19:53:23.880336461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
            correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3981bcd4a5c0>: 
    failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
      "name": "fa758444-9da4-4261-b65a-fb9af0416aae",
      "status": "Failed",
      "startTime": "2026-10-28T20:11:05.048147318Z",
      "endTime": "2026-10-28T20:40:15.036753709Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
            correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x3981bcd4a5c0>: 
    failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
      "name": "fa758444-9da4-4261-b65a-fb9af0416aae",
      "status": "Failed",
      "startTime": "2026-10-28T20:11:05.048147318Z",
      "endTime": "2026-10-28T20:40:15.036753709Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
            correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0x9790da22600>: 
    failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
      "name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
      "status": "Failed",
      "startTime": "2026-10-28T20:55:25.009646036Z",
      "endTime": "2026-10-28T21:24:30.625209621Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
            correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0x9790da22600>: 
    failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
      "name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
      "status": "Failed",
      "startTime": "2026-10-28T20:55:25.009646036Z",
      "endTime": "2026-10-28T21:24:30.625209621Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
            correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
            client request ID: unavailable
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x16967f5def20>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
      "name": "54478daa-47e5-4530-91c2-342cc29a1749",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:21.628677591Z",
      "endTime": "2026-10-28T19:53:23.838328082Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
            correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x16967f5def20>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
      "name": "54478daa-47e5-4530-91c2-342cc29a1749",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:21.628677591Z",
      "endTime": "2026-10-28T19:53:23.838328082Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
            correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x3d00e53ba4e0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
      "name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:17.131948207Z",
      "endTime": "2026-10-28T19:53:23.815769706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
            correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x3d00e53ba4e0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
      "name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
      "status": "Failed",
      "startTime": "2026-10-28T19:24:17.131948207Z",
      "endTime": "2026-10-28T19:53:23.815769706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
            correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0x1c6ab1a4a4c0>: 
    failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
      "name": "13f26b26-2324-48a4-82df-d8b324b13996",
      "status": "Failed",
      "startTime": "2026-10-28T20:52:21.783931077Z",
      "endTime": "2026-10-28T21:21:30.723052246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
            correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0x1c6ab1a4a4c0>: 
    failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
      "name": "13f26b26-2324-48a4-82df-d8b324b13996",
      "status": "Failed",
      "startTime": "2026-10-28T20:52:21.783931077Z",
      "endTime": "2026-10-28T21:21:30.723052246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
            correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
            client request ID: unavailable
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x26d436e0e5e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rgpk9lmtvnvv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105",
      "name": "357cbb72-4fad-4dd1-860d-138b89ef8105",
      "status": "Failed",
      "startTime": "2026-10-28T23:13:44.545799939Z",
      "endTime": "2026-10-28T23:32:49.563451274Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pkf64-8lv9b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pkf64-8lv9b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
            correlation ID: 8689aa1d-b711-4817-af27-601b622fb10c
            client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0x26d436e0e5e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rgpk9lmtvnvv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105",
      "name": "357cbb72-4fad-4dd1-860d-138b89ef8105",
      "status": "Failed",
      "startTime": "2026-10-28T23:13:44.545799939Z",
      "endTime": "2026-10-28T23:32:49.563451274Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pkf64-8lv9b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pkf64-8lv9b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
            correlation ID: 8689aa1d-b711-4817-af27-601b622fb10c
            client request ID: unavailable
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115579573927153664
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
full failure pattern:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x2f7afefabce0>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2f7afec3aa20>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2f7afec3aa00>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2f7aff732b40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2f7afee8d220>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2f7afee8d200>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2f7afee1fae0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
    <*errors.joinError | 0x2f7afefabce0>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2f7afec3aa20>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2f7afec3aa00>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2f7aff732b40>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2f7afee8d220>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2f7afee8d200>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2f7afee1fae0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2114854607891992576
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x161cee87c740>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
      "name": "aa729d08-abec-481d-a5d5-3a517e315df9",
      "status": "Failed",
      "startTime": "2026-10-27T18:45:35.525302675Z",
      "endTime": "2026-10-27T19:04:37.944268372Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
            correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x161cee87c740>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
      "name": "aa729d08-abec-481d-a5d5-3a517e315df9",
      "status": "Failed",
      "startTime": "2026-10-27T18:45:35.525302675Z",
      "endTime": "2026-10-27T19:04:37.944268372Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
            correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115142680974462976
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xd2fa34b2580>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
      "name": "e56720ec-9cdb-4896-bffe-b107388330fa",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:26.535304826Z",
      "endTime": "2026-10-28T15:50:27.739063015Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
            correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xd2fa34b2580>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
      "name": "e56720ec-9cdb-4896-bffe-b107388330fa",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:26.535304826Z",
      "endTime": "2026-10-28T15:50:27.739063015Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
            correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x320c1cc145a0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
      "name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
      "status": "Failed",
      "startTime": "2026-10-28T13:43:08.173393078Z",
      "endTime": "2026-10-28T14:12:10.809270873Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
            correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x320c1cc145a0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
      "name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
      "status": "Failed",
      "startTime": "2026-10-28T13:43:08.173393078Z",
      "endTime": "2026-10-28T14:12:10.809270873Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
            correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0x2df91d94ebc0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
      "name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
      "status": "Failed",
      "startTime": "2026-10-28T19:52:14.429975606Z",
      "endTime": "2026-10-28T20:21:16.149275515Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
            correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0x2df91d94ebc0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
      "name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
      "status": "Failed",
      "startTime": "2026-10-28T19:52:14.429975606Z",
      "endTime": "2026-10-28T20:21:16.149275515Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
            correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x144b1f0220a8>: 
    resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
      "name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
      "status": "Failed",
      "startTime": "2026-10-28T14:05:17.896484451Z",
      "endTime": "2026-10-28T14:29:22.39155595Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
            correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x144b1f0220a8>: 
    resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
      "name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
      "status": "Failed",
      "startTime": "2026-10-28T14:05:17.896484451Z",
      "endTime": "2026-10-28T14:29:22.39155595Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
            correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x33d30568cba0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
      "name": "30b1eeed-8a58-491f-a679-903582bd2428",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:06.381277659Z",
      "endTime": "2026-10-28T15:50:07.62996819Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
            correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x33d30568cba0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
      "name": "30b1eeed-8a58-491f-a679-903582bd2428",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:06.381277659Z",
      "endTime": "2026-10-28T15:50:07.62996819Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
            correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x219ad233aca0>: 
    failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
      "name": "e768a2f1-bf75-4afe-9acf-741914b73091",
      "status": "Failed",
      "startTime": "2026-10-28T20:11:31.970768227Z",
      "endTime": "2026-10-28T20:40:35.054714998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
            correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x219ad233aca0>: 
    failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
      "name": "e768a2f1-bf75-4afe-9acf-741914b73091",
      "status": "Failed",
      "startTime": "2026-10-28T20:11:31.970768227Z",
      "endTime": "2026-10-28T20:40:35.054714998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
            correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, ignition-server-proxy, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1c9f11d305c0>: 
    failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
      "name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
      "status": "Failed",
      "startTime": "2026-10-28T20:57:10.002650956Z",
      "endTime": "2026-10-28T21:26:10.63929862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
            correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x1c9f11d305c0>: 
    failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
      "name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
      "status": "Failed",
      "startTime": "2026-10-28T20:57:10.002650956Z",
      "endTime": "2026-10-28T21:26:10.63929862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
            correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-controller-manager, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x3da3f19209c0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
      "name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:36.875741431Z",
      "endTime": "2026-10-28T20:43:44.887394978Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
            correlation ID: 84abeaa9-4051-418b-b066-32246401955a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x3da3f19209c0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
      "name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
      "status": "Failed",
      "startTime": "2026-10-28T20:14:36.875741431Z",
      "endTime": "2026-10-28T20:43:44.887394978Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
            correlation ID: 84abeaa9-4051-418b-b066-32246401955a
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
    <*fmt.wrapError | 0x257516f84460>: 
    failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
      "name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:31.746407548Z",
      "endTime": "2026-10-28T15:50:37.764425706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
            correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
    <*fmt.wrapError | 0x257516f84460>: 
    failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
      "name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:31.746407548Z",
      "endTime": "2026-10-28T15:50:37.764425706Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
            correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0x2840a3490800>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
      "name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
      "status": "Failed",
      "startTime": "2026-10-28T20:48:29.562853635Z",
      "endTime": "2026-10-28T21:17:30.601011159Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
            correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0x2840a3490800>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
      "name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
      "status": "Failed",
      "startTime": "2026-10-28T20:48:29.562853635Z",
      "endTime": "2026-10-28T21:17:30.601011159Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
            correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x3290e061a680>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
      "name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
      "status": "Failed",
      "startTime": "2026-10-28T20:57:27.436526224Z",
      "endTime": "2026-10-28T21:26:30.622804028Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
            correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x3290e061a680>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
      "name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
      "status": "Failed",
      "startTime": "2026-10-28T20:57:27.436526224Z",
      "endTime": "2026-10-28T21:26:30.622804028Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
            correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x1d947782ea20>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
      "name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
      "status": "Failed",
      "startTime": "2026-10-28T20:01:08.393167298Z",
      "endTime": "2026-10-28T20:30:16.146752142Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
            correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0x1d947782ea20>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
      "name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
      "status": "Failed",
      "startTime": "2026-10-28T20:01:08.393167298Z",
      "endTime": "2026-10-28T20:30:16.146752142Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
            correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x2f2f6cd22960>: 
    failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
      "name": "67af219e-4e40-4a49-ae77-1d482039a909",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:37.478622401Z",
      "endTime": "2026-10-28T15:50:37.723667825Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
            correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
    <*fmt.wrapError | 0x2f2f6cd22960>: 
    failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
      "name": "67af219e-4e40-4a49-ae77-1d482039a909",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:37.478622401Z",
      "endTime": "2026-10-28T15:50:37.723667825Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
            correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x28ad82ebac20>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
      "name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
      "status": "Failed",
      "startTime": "2026-10-28T20:01:11.67931549Z",
      "endTime": "2026-10-28T20:30:16.178438369Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
            correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x28ad82ebac20>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
      "name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
      "status": "Failed",
      "startTime": "2026-10-28T20:01:11.67931549Z",
      "endTime": "2026-10-28T20:30:16.178438369Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
            correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x3d893d926940>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
      "name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
      "status": "Failed",
      "startTime": "2026-10-28T14:39:20.447678223Z",
      "endTime": "2026-10-28T15:08:28.688541915Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
            correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0x3d893d926940>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
      "name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
      "status": "Failed",
      "startTime": "2026-10-28T14:39:20.447678223Z",
      "endTime": "2026-10-28T15:08:28.688541915Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
            correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, kube-controller-manager, kube-scheduler]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x186bf96476a0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
      "name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
      "status": "Failed",
      "startTime": "2026-10-28T13:42:56.996297525Z",
      "endTime": "2026-10-28T14:12:00.914226401Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
            correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x186bf96476a0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
      "name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
      "status": "Failed",
      "startTime": "2026-10-28T13:42:56.996297525Z",
      "endTime": "2026-10-28T14:12:00.914226401Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
            correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x1308b97a65a0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
      "name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
      "status": "Failed",
      "startTime": "2026-10-28T20:58:09.127282806Z",
      "endTime": "2026-10-28T21:27:10.682711777Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
            correlation ID: 49b88924-064f-4f52-9480-109cf607a390
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x1308b97a65a0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
      "name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
      "status": "Failed",
      "startTime": "2026-10-28T20:58:09.127282806Z",
      "endTime": "2026-10-28T21:27:10.682711777Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
            correlation ID: 49b88924-064f-4f52-9480-109cf607a390
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x7c22833c460>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
      "name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:06.28860999Z",
      "endTime": "2026-10-28T15:50:07.63308505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
            correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x7c22833c460>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
      "name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
      "status": "Failed",
      "startTime": "2026-10-28T15:21:06.28860999Z",
      "endTime": "2026-10-28T15:50:07.63308505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
            correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115430933703168000
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
    <*fmt.wrapError | 0xd7eab0d0620>: 
    failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
      "name": "36728157-067b-4375-a959-9c9ce43e534b",
      "status": "Failed",
      "startTime": "2026-10-28T20:56:26.269135944Z",
      "endTime": "2026-10-28T21:25:30.687402107Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
            correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
    <*fmt.wrapError | 0xd7eab0d0620>: 
    failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
      "name": "36728157-067b-4375-a959-9c9ce43e534b",
      "status": "Failed",
      "startTime": "2026-10-28T20:56:26.269135944Z",
      "endTime": "2026-10-28T21:25:30.687402107Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
            correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2115519015970410496
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: interfaces creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.96%1 of 51 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x27665d95f6c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
      "name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
      "status": "Failed",
      "startTime": "2026-10-27T03:14:20.369455888Z",
      "endTime": "2026-10-27T03:33:20.394901036Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
            correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x27665d95f6c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
      "name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
      "status": "Failed",
      "startTime": "2026-10-27T03:14:20.369455888Z",
      "endTime": "2026-10-27T03:33:20.394901036Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
            correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2114899906912063488