CIHealth
<
Single Day: Oct 25▾
Last 7 Days Weekly: Oct 19 - Oct 25 Sprint: Oct 12 - Oct 25 Single Day: Oct 25
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 30 · Failures matched: 45 · Total runs: 17 · Runs affected: 13

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3f211b784120>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fqjfphzpbt4x" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59",
      "name": "dc5b2708-f9a2-4d86-915e-eb8ba91cbd59",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:22.163428437Z",
      "endTime": "2026-10-25T11:30:25.784218117Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.19 is Partial (want Completed), started 19m44s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59
            correlation ID: 16b92b93-ef61-4f4d-9bbb-a63f0c7ebf4f
            client request ID: cc5cc315-91ce-4032-81e3-c0e49c80e860
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3f211b784120>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fqjfphzpbt4x" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59",
      "name": "dc5b2708-f9a2-4d86-915e-eb8ba91cbd59",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:22.163428437Z",
      "endTime": "2026-10-25T11:30:25.784218117Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.19 is Partial (want Completed), started 19m44s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59
            correlation ID: 16b92b93-ef61-4f4d-9bbb-a63f0c7ebf4f
            client request ID: cc5cc315-91ce-4032-81e3-c0e49c80e860
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x299d616e8140>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-bcvsvbp5d8t7" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671",
      "name": "f4d6594c-1ee5-4e01-824a-b335b8b1a671",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:32.174992685Z",
      "endTime": "2026-10-25T11:30:36.037388523Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m56s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671
            correlation ID: 88e2d1ef-c0a9-47ed-aaa6-0693add21a52
            client request ID: 8ab0d68c-218c-40c0-b734-d3980ba8ecab
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x299d616e8140>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-bcvsvbp5d8t7" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671",
      "name": "f4d6594c-1ee5-4e01-824a-b335b8b1a671",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:32.174992685Z",
      "endTime": "2026-10-25T11:30:36.037388523Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m56s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671
            correlation ID: 88e2d1ef-c0a9-47ed-aaa6-0693add21a52
            client request ID: 8ab0d68c-218c-40c0-b734-d3980ba8ecab
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-zgspqd" with version 4.22.18 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x1120db708700>: 
    failed to create HCP cluster cluster-zstream-4-22-zgspqd: failed waiting for cluster="cluster-zstream-4-22-zgspqd" in resourcegroup="rg-zstream-upgrade-4-22-4qv6w5xvr977" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e",
      "name": "ffe95342-e02c-4496-883e-5b1bf611d99e",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:19.965849634Z",
      "endTime": "2026-10-25T11:30:25.598823992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, olm-operator, capi-provider, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, redhat-operators-catalog, openshift-route-controller-manager, community-operators-catalog, certified-operators-catalog, cluster-storage-operator, catalog-operator, openshift-controller-manager, cluster-version-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e
            correlation ID: e700165f-ec9f-48e4-abf2-18b14bbd9d95
            client request ID: cdbafbf0-d5e0-4ed7-8b28-b50e4bf8f6b8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-zgspqd" with version 4.22.18 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x1120db708700>: 
    failed to create HCP cluster cluster-zstream-4-22-zgspqd: failed waiting for cluster="cluster-zstream-4-22-zgspqd" in resourcegroup="rg-zstream-upgrade-4-22-4qv6w5xvr977" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e",
      "name": "ffe95342-e02c-4496-883e-5b1bf611d99e",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:19.965849634Z",
      "endTime": "2026-10-25T11:30:25.598823992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, olm-operator, capi-provider, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, redhat-operators-catalog, openshift-route-controller-manager, community-operators-catalog, certified-operators-catalog, cluster-storage-operator, catalog-operator, openshift-controller-manager, cluster-version-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e
            correlation ID: e700165f-ec9f-48e4-abf2-18b14bbd9d95
            client request ID: cdbafbf0-d5e0-4ed7-8b28-b50e4bf8f6b8
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x105dbf764180>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mv5sq4l7jv4r" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b",
      "name": "35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:21.658859222Z",
      "endTime": "2026-10-25T11:30:25.924003842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: capi-provider; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b
            correlation ID: d887bb88-655e-4cf4-84f8-c59ec1d133f4
            client request ID: 066a4e25-cc0e-4d6f-a9c5-8e71913c3b6c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x105dbf764180>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mv5sq4l7jv4r" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b",
      "name": "35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:21.658859222Z",
      "endTime": "2026-10-25T11:30:25.924003842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: capi-provider; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b
            correlation ID: d887bb88-655e-4cf4-84f8-c59ec1d133f4
            client request ID: 066a4e25-cc0e-4d6f-a9c5-8e71913c3b6c
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x1afe2fabc320>: 
    failed to create HCP cluster ea-list7tqb74: failed waiting for cluster="ea-list7tqb74" in resourcegroup="ea-list-8mbx8mstb22c" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41",
      "name": "506cd809-e6bc-45d8-aa3a-45042c374b41",
      "status": "Failed",
      "startTime": "2026-10-25T11:02:00.137646561Z",
      "endTime": "2026-10-25T11:31:06.609068276Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, community-operators-catalog, packageserver, cluster-node-tuning-operator, cluster-storage-operator, ingress-operator, dns-operator, catalog-operator, cluster-network-operator, cluster-version-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, certified-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, cluster-policy-controller, olm-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41
            correlation ID: a86e7a8c-dce2-49be-8312-2dfeeebedc58
            client request ID: 03f569a8-daf7-4041-b81b-1efe61e6782e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x1afe2fabc320>: 
    failed to create HCP cluster ea-list7tqb74: failed waiting for cluster="ea-list7tqb74" in resourcegroup="ea-list-8mbx8mstb22c" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41",
      "name": "506cd809-e6bc-45d8-aa3a-45042c374b41",
      "status": "Failed",
      "startTime": "2026-10-25T11:02:00.137646561Z",
      "endTime": "2026-10-25T11:31:06.609068276Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, community-operators-catalog, packageserver, cluster-node-tuning-operator, cluster-storage-operator, ingress-operator, dns-operator, catalog-operator, cluster-network-operator, cluster-version-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, certified-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, cluster-policy-controller, olm-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41
            correlation ID: a86e7a8c-dce2-49be-8312-2dfeeebedc58
            client request ID: 03f569a8-daf7-4041-b81b-1efe61e6782e
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-vfvmsnwsd5sg/cluster-candidate-5-0-lh7h4v should provision
Unexpected error:
    <*fmt.wrapError | 0x274288768300>: 
    failed to create HCP cluster cluster-candidate-5-0-lh7h4v: failed waiting for cluster="cluster-candidate-5-0-lh7h4v" in resourcegroup="rg-candidate-5-0-vfvmsnwsd5sg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8",
      "name": "f8664d31-ed00-44b7-89f1-f9abec4ea6b8",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:22.79701272Z",
      "endTime": "2026-10-25T11:30:25.820223913Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 20m56s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8
            correlation ID: d572d016-0277-4113-8a96-1da300eb89da
            client request ID: a981c6e6-813e-4373-81f7-4aab2cf9a886
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-vfvmsnwsd5sg/cluster-candidate-5-0-lh7h4v should provision
Unexpected error:
    <*fmt.wrapError | 0x274288768300>: 
    failed to create HCP cluster cluster-candidate-5-0-lh7h4v: failed waiting for cluster="cluster-candidate-5-0-lh7h4v" in resourcegroup="rg-candidate-5-0-vfvmsnwsd5sg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8",
      "name": "f8664d31-ed00-44b7-89f1-f9abec4ea6b8",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:22.79701272Z",
      "endTime": "2026-10-25T11:30:25.820223913Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 20m56s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8
            correlation ID: d572d016-0277-4113-8a96-1da300eb89da
            client request ID: a981c6e6-813e-4373-81f7-4aab2cf9a886
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailab...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x2a59ed8de2a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-xf7h6gdhg87b" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b",
      "name": "0c13cc8b-efe6-4584-9ae2-571f7f85e94b",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:20.835629052Z",
      "endTime": "2026-10-25T11:30:25.907970891Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b
            correlation ID: f5a00756-4d2f-4428-b3ea-e6cf5e3ef0a1
            client request ID: 03254b25-a5c1-4e3b-88dd-5aeb8bec557c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
    <*fmt.wrapError | 0x2a59ed8de2a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-xf7h6gdhg87b" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b",
      "name": "0c13cc8b-efe6-4584-9ae2-571f7f85e94b",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:20.835629052Z",
      "endTime": "2026-10-25T11:30:25.907970891Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b
            correlation ID: f5a00756-4d2f-4428-b3ea-e6cf5e3ef0a1
            client request ID: 03254b25-a5c1-4e3b-88dd-5aeb8bec557c
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rdkvkj' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...
full failure pattern:
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rdkvkj' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cprxqg"
Unexpected error:
    <*errors.joinError | 0x30c853d502b8>: 
    VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-rdkvkj" not yet present in zone rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/*.e2e-dns-rdkvkj
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-rdkvkj' does not exist in resource group 'rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/*.e2e-dns-rdkvkj
            correlation ID: 984d9387-98d9-461e-8f5b-657273ddef57
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cprxqg"
Unexpected error:
    <*errors.joinError | 0x30c853d502b8>: 
    VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-rdkvkj" not yet present in zone rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/*.e2e-dns-rdkvkj
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-rdkvkj' does not exist in resource group 'rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/*.e2e-dns-rdkvkj
            correlation ID: 984d9387-98d9-461e-8f5b-657273ddef57
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #77392114494146650574848
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] ClusterCredentialMetricMissing fired
full failure pattern:
alert [svc] ClusterCredentialMetricMissing fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-25T11:11:08Z
  Severity:    Sev3
  Labels:      alertname="ClusterCredentialMetricMissing", cluster="ci01-j1877504-svc", severity="warning"
  Description: Credential refresher expiration metric is missing for service cluster ci01-j1877504-svc.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ClusterCredentialMetricMissing does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #60812114294657641877504
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] FluentBitIngestionPaused fired
full failure pattern:
alert [svc] FluentBitIngestionPaused fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T11:48:54Z
  Ended:       2026-10-25T11:58:53Z
  Severity:    Sev3
  Labels:      alertname="FluentBitIngestionPaused", cluster="ci01-j7524864-mgmt-2", component="monitoring-infrastructure", pod="arobit-forwarder-q4jd5", region="westus3", severity="warning"
  Description: Fluent Bit pod arobit-forwarder-q4jd5 on cluster ci01-j7524864-mgmt-2 has paused collecting new log data for at least 5 minutes.
Ingestion pauses when Fluent Bit\'s internal memory or storage buffers are full, typically caused by backpressure from a slow or failing output.
Investigate the Fluent Bit logs for the specific error details and check the Kusto instance health.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FluentBitIngestionPaused does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRmissing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0
full failure pattern:
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0x2f3433d553d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0x2f3433d553d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #77122114294348290985984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRmissing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0
full failure pattern:
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 620.639s.
Expected success, but got an error:
    <*errors.errorString | 0x1cfcad623430>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 620.639s.
Expected success, but got an error:
    <*errors.errorString | 0x1cfcad623430>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #77122114333714598072320
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRmissing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0
full failure pattern:
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e15.88%1 of 17 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 630.763s.
Expected success, but got an error:
    <*errors.errorString | 0x2d6e687aea00>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 630.763s.
Expected success, but got an error:
    <*errors.errorString | 0x2d6e687aea00>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #77122114409269653671936
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert211.76%2 of 17 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 19: 5 · Oct 20: 8 · Oct 21: 0 · Oct 22: 2 · Oct 23: 3 · Oct 24: 0 · Oct 25: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T13:57:52Z
  Ended:       2026-10-25T14:23:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b19b2a4f-abec-4efb-8fa1-a0f49de6f865"
  Description: Pod ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-25T13:57:52Z
  Ended:       2026-10-25T14:23:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b19b2a4f-abec-4efb-8fa1-a0f49de6f865"
  Description: Pod ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T11:41:40Z
  Ended:       2026-10-25T11:51:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le", pod="capi-provider-6d7bfdf44-24wm6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5602ef00-22ff-45dd-b3cb-39a5ba53e966"
  Description: Pod ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le/capi-provider-6d7bfdf44-24wm6 (manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-25T11:41:40Z
  Ended:       2026-10-25T11:51:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le", pod="capi-provider-6d7bfdf44-24wm6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5602ef00-22ff-45dd-b3cb-39a5ba53e966"
  Description: Pod ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le/capi-provider-6d7bfdf44-24wm6 (manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Resolved
  Started:     2026-10-25T11:43:40Z
  Ended:       2026-10-25T11:51:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp733dhbjgega0lnvcjvgg1grv9tf4-i6w1d6x7y2t6h2v", pod="capi-provider-8f7b799d8-49zjj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="706780f8-3192-4947-b780-05f86e3d1b53"
  Description: Pod ocm-arohcpci01-2tlp733dhbjgega0lnvcjvgg1grv9tf4-i6w1d6x7y2t6h2v/capi-provider-8f7b799d8-49zjj (manager) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #61912114333668922101760
PR #61912114294563647524864
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert15.88%1 of 17 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsOct 19: 13 · Oct 20: 8 · Oct 21: 11 · Oct 22: 4 · Oct 23: 3 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
alert fired 34 time(s) (3 unknown, 31 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T11:42:42Z
  Ended:       2026-10-25T11:50:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl", pod="azure-file-csi-driver-controller-66f6c89876-l6fts", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl/azure-file-csi-driver-controller-66f6c89876-l6fts has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-25T11:44:40Z
  Ended:       2026-10-25T11:50:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl", pod="multus-admission-controller-58547c8cf6-lmftl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl/multus-admission-controller-58547c8cf6-lmftl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-25T11:53:40Z
  Ended:       2026-10-25T11:59:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp70no771drpm6jf2csmj08ibs461h-s2r6f9c0k1q1q9w", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tlp70no771drpm6jf2csmj08ibs461h-s2r6f9c0k1q1q9w/etcd-0 has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert15.88%1 of 17 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsOct 19: 4 · Oct 20: 9 · Oct 21: 1 · Oct 22: 3 · Oct 23: 3 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T14:09:53Z
  Ended:       2026-10-25T14:23:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-25T14:10:54Z
  Ended:       2026-10-25T14:19:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114333668922101760
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
e2e15.88%1 of 17 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 19: 5 · Oct 20: 8 · Oct 21: 0 · Oct 22: 3 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x209528ec2340>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ktmjg5k5mr4p" to finish creating: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564",
      "name": "cd12106f-dc74-4bb2-88d9-552815761564",
      "status": "Failed",
      "startTime": "2026-10-25T13:33:58.83934802Z",
      "endTime": "2026-10-25T14:03:01.452063547Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
            correlation ID: a999ca91-1f04-4667-8416-be09ce537e26
            client request ID: e9d482d5-6b2f-4cbd-99d8-1b5bb0124d29
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x209528ec2340>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ktmjg5k5mr4p" to finish creating: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564",
      "name": "cd12106f-dc74-4bb2-88d9-552815761564",
      "status": "Failed",
      "startTime": "2026-10-25T13:33:58.83934802Z",
      "endTime": "2026-10-25T14:03:01.452063547Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
            correlation ID: a999ca91-1f04-4667-8416-be09ce537e26
            client request ID: e9d482d5-6b2f-4cbd-99d8-1b5bb0124d29
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114333668922101760
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert15.88%1 of 17 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 19: 5 · Oct 20: 8 · Oct 21: 0 · Oct 22: 2 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T13:55:59Z
  Ended:       2026-10-25T14:19:54Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-25T14:06:53Z
  Ended:       2026-10-25T14:19:54Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114333668922101760
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e15.88%1 of 17 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsOct 19: 2 · Oct 20: 6 · Oct 21: 2 · Oct 22: 0 · Oct 23: 5 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x34e2e46f8680>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mfmrpz5xbrdj" to finish creating: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14",
      "name": "3d1a2520-89cf-4c42-a91c-ec725b89af14",
      "status": "Failed",
      "startTime": "2026-10-25T11:04:09.039925979Z",
      "endTime": "2026-10-25T11:23:09.310567948Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
            correlation ID: 9acdbad6-f09a-46a0-a926-c738c62f3c07
            client request ID: 916943ff-0406-46da-9830-2889e6355ff3
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x34e2e46f8680>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mfmrpz5xbrdj" to finish creating: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14",
      "name": "3d1a2520-89cf-4c42-a91c-ec725b89af14",
      "status": "Failed",
      "startTime": "2026-10-25T11:04:09.039925979Z",
      "endTime": "2026-10-25T11:23:09.310567948Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
            correlation ID: 9acdbad6-f09a-46a0-a926-c738c62f3c07
            client request ID: 916943ff-0406-46da-9830-2889e6355ff3
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #60812114294657641877504
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert15.88%1 of 17 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 6 · Oct 23: 1 · Oct 24: 1 · Oct 25: 1none
Full failure examples (1)
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T19:03:55Z
  Ended:       2026-10-25T19:10:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-25T19:03:55Z
  Ended:       2026-10-25T19:09:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-25T19:03:55Z
  Ended:       2026-10-25T19:10:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #77122114409269653671936
alert [svc] KubePodCrashLooping fired
full failure pattern:
alert [svc] KubePodCrashLooping fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history
alert423.53%4 of 17 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior historyShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 2 · Oct 22: 6 · Oct 23: 3 · Oct 24: 2 · Oct 25: 4none
Full failure examples (3)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-25T18:47:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-bfwrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="97b76fc8-3f29-44d9-83c4-855c4cca2cf8"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-bfwrz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-25T18:47:54Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.215:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mftk6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c4f6f9ce-1370-407b-a328-72082e9577a0"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mftk6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-25T18:50:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.215:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mftk6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c4f6f9ce-1370-407b-a328-72082e9577a0"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mftk6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-25T18:50:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-bfwrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="97b76fc8-3f29-44d9-83c4-855c4cca2cf8"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-bfwrz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-25T13:52:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdq5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9c6d2039-7988-4be2-8890-d290ed7d2d20"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdq5f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-25T13:54:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdq5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9c6d2039-7988-4be2-8890-d290ed7d2d20"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdq5f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-25T13:55:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.141:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-h6zb9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8720bf8b-027c-410b-9181-eafea5b14401"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-h6zb9 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-25T13:55:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.141:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-h6zb9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8720bf8b-027c-410b-9181-eafea5b14401"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-h6zb9 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
alert fired 8 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-25T11:31:30Z
  Ended:       2026-10-25T11:58:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-czwqq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="76e86419-4829-44c0-a136-817aadd188d4"
  Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-czwqq (manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-25T11:31:30Z
  Ended:       2026-10-25T11:58:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-czwqq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="76e86419-4829-44c0-a136-817aadd188d4"
  Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-czwqq (manager) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Resolved
  Started:     2026-10-25T11:37:32Z
  Ended:       2026-10-25T11:48:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-67tll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5593f36-ff37-4a56-9501-050ee9a5f489"
  Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-67tll (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Resolved
  Started:     2026-10-25T11:37:32Z
  Ended:       2026-10-25T11:48:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-67tll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5593f36-ff37-4a56-9501-050ee9a5f489"
  Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-67tll (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Resolved
  Started:     2026-10-25T11:39:32Z
  Ended:       2026-10-25T11:48:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-j5hzx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="eabaed55-b955-4173-b78d-fec0366c15f2"
  Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-j5hzx (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Resolved
  Started:     2026-10-25T11:39:32Z
  Ended:       2026-10-25T11:48:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-j5hzx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="eabaed55-b955-4173-b78d-fec0366c15f2"
  Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-j5hzx (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Resolved
  Started:     2026-10-25T11:41:31Z
  Ended:       2026-10-25T11:52:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="ocm-controller", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="ocm-controller-859484fc49-pj8f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a3ea24fa-145a-42ea-812b-1fe46f10bfd1"
  Description: Pod multicluster-engine/ocm-controller-859484fc49-pj8f8 (ocm-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Resolved
  Started:     2026-10-25T11:41:31Z
  Ended:       2026-10-25T11:52:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="ocm-controller", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="ocm-controller-859484fc49-pj8f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a3ea24fa-145a-42ea-812b-1fe46f10bfd1"
  Description: Pod multicluster-engine/ocm-controller-859484fc49-pj8f8 (ocm-controller) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #77122114409269653671936
PR #77122114333714598072320
PR #61912114294563647524864
PR #77122114294348290985984
failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context de...
full failure pattern:
failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision423.53%4 of 17 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 4none
Full failure examples (3)
time=2026-10-25T10:27:22.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n"
time=2026-10-25T10:27:22.644Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]"
time=2026-10-25T10:27:22.644Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2
time=2026-10-25T10:27:22.918Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller
time=2026-10-25T10:27:23.279Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false
time=2026-10-25T10:27:23.279Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2
time=2026-10-25T10:27:23.445Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2
time=2026-10-25T10:32:28.984Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-25T10:32:28.985Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-25T08:14:44.299Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-25T08:14:45.051Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]"
time=2026-10-25T08:14:45.051Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1
time=2026-10-25T08:14:45.565Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller
time=2026-10-25T08:14:45.984Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-25T08:14:45.984Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1
time=2026-10-25T08:14:46.156Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1
time=2026-10-25T08:19:54.254Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-25T08:19:54.254Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-25T06:58:33.435Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-25T06:58:33.964Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]"
time=2026-10-25T06:58:33.964Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1
time=2026-10-25T06:58:34.148Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller
time=2026-10-25T06:58:34.349Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-25T06:58:34.350Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1
time=2026-10-25T06:58:34.404Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1
time=2026-10-25T07:03:38.507Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-25T07:03:38.507Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=1)3
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=2)1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #76622114287994985779200
PR #76622114257867568582656
PR #76622114242927562592256
PR #76622114223810998702080
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e211.76%2 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xd75a95e43e0>: 
    DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xd75a8fba020>{
            msg: "DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xd75a95e43e0>: 
    DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xd75a8fba020>{
            msg: "DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc4f5ed30b40>: 
    DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xc4f5f2da4a0>{
            msg: "DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc4f5ed30b40>: 
    DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0xc4f5f2da4a0>{
            msg: "DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
PR #77122114409269653671936
PR #77122114294348290985984
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-q5hjsn" with version 4.20.42 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x18f75457b1e0>: 
    failed to create HCP cluster cluster-zstream-4-20-q5hjsn: failed waiting for cluster="cluster-zstream-4-20-q5hjsn" in resourcegroup="rg-zstream-upgrade-4-20-jb8s5lbcxszk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5",
      "name": "5400fbc4-9a5a-4a07-8b68-53299b9929f5",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:30.073248883Z",
      "endTime": "2026-10-25T11:30:35.871032188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.42 is Partial (want Completed), started 23m47s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5
            correlation ID: cf35d619-4e23-43fc-8aeb-ee92df9074e6
            client request ID: 68e2030f-31dd-4592-871e-2e8df326a35b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-q5hjsn" with version 4.20.42 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x18f75457b1e0>: 
    failed to create HCP cluster cluster-zstream-4-20-q5hjsn: failed waiting for cluster="cluster-zstream-4-20-q5hjsn" in resourcegroup="rg-zstream-upgrade-4-20-jb8s5lbcxszk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5",
      "name": "5400fbc4-9a5a-4a07-8b68-53299b9929f5",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:30.073248883Z",
      "endTime": "2026-10-25T11:30:35.871032188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.42 is Partial (want Completed), started 23m47s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5
            correlation ID: cf35d619-4e23-43fc-8aeb-ee92df9074e6
            client request ID: 68e2030f-31dd-4592-871e-2e8df326a35b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x134cca4de620>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-smqlw28gpmfk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2",
      "name": "d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:59.115997025Z",
      "endTime": "2026-10-25T11:31:07.16366343Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m52s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2
            correlation ID: 9f778ec9-bb64-4e1c-9a2f-a32317e4bb4a
            client request ID: bfadd86d-0c2c-4751-a28f-5658e7a124e6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x134cca4de620>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-smqlw28gpmfk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2",
      "name": "d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:59.115997025Z",
      "endTime": "2026-10-25T11:31:07.16366343Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m52s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2
            correlation ID: 9f778ec9-bb64-4e1c-9a2f-a32317e4bb4a
            client request ID: bfadd86d-0c2c-4751-a28f-5658e7a124e6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x2b59459fc100>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-t7spx49mgg2h" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38",
      "name": "fde9935e-6ef0-4518-bbc5-60b5d4f16d38",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:26.130044072Z",
      "endTime": "2026-10-25T11:30:36.059795656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 22m37s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38
            correlation ID: 1127ed99-2d51-4906-b522-5ba778a6c853
            client request ID: eb7f7962-a820-4a0a-acaa-426a2d0c04e3
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x2b59459fc100>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-t7spx49mgg2h" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38",
      "name": "fde9935e-6ef0-4518-bbc5-60b5d4f16d38",
      "status": "Failed",
      "startTime": "2026-10-25T11:01:26.130044072Z",
      "endTime": "2026-10-25T11:30:36.059795656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 22m37s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38
            correlation ID: 1127ed99-2d51-4906-b522-5ba778a6c853
            client request ID: eb7f7962-a820-4a0a-acaa-426a2d0c04e3
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining ...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x381116f3a0c0>: 
    resource group "admin-api-serialconsole-bootdiag-l9k6ftz85vvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-l9k6ftz85vvb" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c",
      "name": "894117ff-70c7-42e1-b1b8-b8df80eb0d8c",
      "status": "Failed",
      "startTime": "2026-10-25T11:28:30.111139221Z",
      "endTime": "2026-10-25T11:52:32.090609688Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c
            correlation ID: d229c55a-e375-4f40-bb2a-f3ad97610736
            client request ID: aa4bc9f4-d5d8-494d-94b7-252189caf6ce
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x381116f3a0c0>: 
    resource group "admin-api-serialconsole-bootdiag-l9k6ftz85vvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-l9k6ftz85vvb" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c",
      "name": "894117ff-70c7-42e1-b1b8-b8df80eb0d8c",
      "status": "Failed",
      "startTime": "2026-10-25T11:28:30.111139221Z",
      "endTime": "2026-10-25T11:52:32.090609688Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c
            correlation ID: d229c55a-e375-4f40-bb2a-f3ad97610736
            client request ID: aa4bc9f4-d5d8-494d-94b7-252189caf6ce
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x22bced7141e0>: 
    resource group "rg-cluster-nsg-subnet-reuse-xcl2892ldqng": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xcl2892ldqng" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a",
      "name": "954cc982-3e76-4b31-9df6-fcc83a76d91a",
      "status": "Failed",
      "startTime": "2026-10-25T11:21:43.829823922Z",
      "endTime": "2026-10-25T11:45:52.860717396Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a
            correlation ID: dc5c263c-66f3-4b74-a013-b67fbdf80f9a
            client request ID: 3eb7aaab-ba2a-414a-b973-c3e5012b0c78
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
    <*errors.joinError | 0x22bced7141e0>: 
    resource group "rg-cluster-nsg-subnet-reuse-xcl2892ldqng": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xcl2892ldqng" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a",
      "name": "954cc982-3e76-4b31-9df6-fcc83a76d91a",
      "status": "Failed",
      "startTime": "2026-10-25T11:21:43.829823922Z",
      "endTime": "2026-10-25T11:45:52.860717396Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a
            correlation ID: dc5c263c-66f3-4b74-a013-b67fbdf80f9a
            client request ID: 3eb7aaab-ba2a-414a-b973-c3e5012b0c78
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x2a5fafb1fdc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dhxpqc2757nk" to finish creating: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415",
      "name": "995ad93d-44a1-4aae-9f3c-cef4914b4415",
      "status": "Failed",
      "startTime": "2026-10-25T11:09:27.17231924Z",
      "endTime": "2026-10-25T11:28:33.403436193Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
            correlation ID: d01faed1-7c94-4749-8892-78ade8753a54
            client request ID: 5d2be0a3-7126-4924-b3ac-5c1c229ea51d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0x2a5fafb1fdc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dhxpqc2757nk" to finish creating: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415",
      "name": "995ad93d-44a1-4aae-9f3c-cef4914b4415",
      "status": "Failed",
      "startTime": "2026-10-25T11:09:27.17231924Z",
      "endTime": "2026-10-25T11:28:33.403436193Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
            correlation ID: d01faed1-7c94-4749-8892-78ade8753a54
            client request ID: 5d2be0a3-7126-4924-b3ac-5c1c229ea51d
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #76572114291888369438720
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s)Show trend detailsOct 19: 3 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x25b4e2d279a0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x25b4e2d279a0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114333668922101760
failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded
full failure pattern:
failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history
provision15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior historyShow trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 4 · Oct 24: 1 · Oct 25: 1none
Full failure examples (1)
time=2026-10-25T11:53:02.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n  Kind: IstioUpgrade\n  AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-25T11:53:04.190Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-25T11:53:04.190Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-25T11:53:05.198Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-25T11:53:05.198Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:53:05.198Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j8850688-svc
time=2026-10-25T11:56:09.852Z level=INFO msg="Cluster provisioning state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade state=Succeeded
time=2026-10-25T11:56:09.852Z level=INFO msg="ARM operation complete" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary elapsed=3m5s
time=2026-10-25T11:56:09.852Z level=INFO msg="Step 1/9: Ensuring MISE ext-authz ConfigMap" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:10.416Z level=INFO msg="ConfigMap created" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/revision-configmap name=istio-shared-configmap-asm-1-30
time=2026-10-25T11:56:10.416Z level=INFO msg="Step 2/9: Checking tag-based namespace safety" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:10.416Z level=INFO msg="Step 3/9: Verifying control plane health and flipping tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.207Z level=INFO msg="Created revision tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/revision-tag webhook=istio-revision-tag-prod-stable-aks-istio-system revision=asm-1-30
time=2026-10-25T11:56:11.207Z level=INFO msg="Step 4/9: Ensuring ingress gateway annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.237Z level=INFO msg="Applying annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/ingress-annotations service=aks-istio-ingressgateway-external
time=2026-10-25T11:56:11.450Z level=INFO msg="Step 5/9: Migrating workloads to target revision" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.510Z level=INFO msg="Step 6/9: Running health check" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Health check passed -- checking for orphaned workloads before completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Step 7/9: Running orphan guard" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="No orphaned workloads -- completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Step 8/9: Completing canary via ARM (point of no return)" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=complete-canary cluster=ci01-j8850688-svc
time=2026-10-25T12:11:14.972Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75972114315309748850688
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 19: 1 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x22535b109360>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-bh6sljsqnwrw" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536",
      "name": "ec3175b8-8979-4854-9d37-2e76e8d09536",
      "status": "Failed",
      "startTime": "2026-10-25T11:02:03.316597788Z",
      "endTime": "2026-10-25T11:31:06.803246105Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, redhat-operators-catalog, redhat-marketplace-catalog, ingress-operator, cluster-image-registry-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, cluster-version-operator, cluster-node-tuning-operator, control-plane-pki-operator, azure-cloud-controller-manager, openshift-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, packageserver, certified-operators-catalog, catalog-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536
            correlation ID: 7d2b87e8-723a-49a9-a4cb-8027a1fb4066
            client request ID: b5b1bd46-74cb-4267-85b1-e1af986392eb
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x22535b109360>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-bh6sljsqnwrw" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536",
      "name": "ec3175b8-8979-4854-9d37-2e76e8d09536",
      "status": "Failed",
      "startTime": "2026-10-25T11:02:03.316597788Z",
      "endTime": "2026-10-25T11:31:06.803246105Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, redhat-operators-catalog, redhat-marketplace-catalog, ingress-operator, cluster-image-registry-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, cluster-version-operator, cluster-node-tuning-operator, control-plane-pki-operator, azure-cloud-controller-manager, openshift-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, packageserver, certified-operators-catalog, catalog-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536
            correlation ID: 7d2b87e8-723a-49a9-a4cb-8027a1fb4066
            client request ID: b5b1bd46-74cb-4267-85b1-e1af986392eb
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 19: 1 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0x2401cda17700>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-6lgfpz" in resourcegroup="rg-np-labels-taints-2pq6dbldqrjk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: f50cf33f-8bc4-4ce0-8a7c-d816cfc4e960",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-bvffjvcy6qo3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997
            correlation ID: 9ef1eaf3-8ba8-425e-8fe5-70f8311af59e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0x2401cda17700>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-6lgfpz" in resourcegroup="rg-np-labels-taints-2pq6dbldqrjk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: f50cf33f-8bc4-4ce0-8a7c-d816cfc4e960",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-bvffjvcy6qo3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997
            correlation ID: 9ef1eaf3-8ba8-425e-8fe5-70f8311af59e
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e15.88%1 of 17 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x3a8a07a1e540>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rvs4gzxj9lgg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399",
      "name": "63403578-fb62-456b-b40e-41ef79e2e399",
      "status": "Failed",
      "startTime": "2026-10-25T11:02:39.014043295Z",
      "endTime": "2026-10-25T11:31:45.790547256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m13s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399
            correlation ID: 38478f64-7ba7-40aa-b720-a7f94d356d8b
            client request ID: 82836f4c-341e-4d1e-88f4-c55b4e681a30
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0x3a8a07a1e540>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rvs4gzxj9lgg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399",
      "name": "63403578-fb62-456b-b40e-41ef79e2e399",
      "status": "Failed",
      "startTime": "2026-10-25T11:02:39.014043295Z",
      "endTime": "2026-10-25T11:31:45.790547256Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m13s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399
            correlation ID: 38478f64-7ba7-40aa-b720-a7f94d356d8b
            client request ID: 82836f4c-341e-4d1e-88f4-c55b4e681a30
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61912114294563647524864