CIHealth
<
Single Day: Sep 28▾
Last 7 Days Weekly: Sep 28 - Oct 4 Sprint: Sep 28 - Oct 11 Single Day: Sep 28
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 68 · Failures matched: 272 · Total runs: 114 · Runs affected: 87

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e21.75%2 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000911660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6qt82pv5cb4k" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088",
      "name": "cd024c78-818b-4b49-8634-2f704d9a8088",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:28.191649845Z",
      "endTime": "2026-09-28T07:47:33.272794005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000911660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6qt82pv5cb4k" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088",
      "name": "cd024c78-818b-4b49-8634-2f704d9a8088",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:28.191649845Z",
      "endTime": "2026-09-28T07:47:33.272794005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001698220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hqr7q7xq4llr" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "name": "e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:08.942854859Z",
      "endTime": "2026-09-28T05:23:14.16072165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001698220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hqr7q7xq4llr" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "name": "e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:08.942854859Z",
      "endTime": "2026-09-28T05:23:14.16072165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0005d2ce0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sbtnmq6dk9ch" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758",
      "name": "1f13de10-f443-42eb-b391-6417a9194758",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:46.663436186Z",
      "endTime": "2026-09-28T05:24:54.296612042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0005d2ce0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sbtnmq6dk9ch" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758",
      "name": "1f13de10-f443-42eb-b391-6417a9194758",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:46.663436186Z",
      "endTime": "2026-09-28T05:24:54.296612042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782104457246334259200
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a6bae0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5w2hj7t2fgmv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "name": "fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:32.768656716Z",
      "endTime": "2026-09-28T13:22:40.579367905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m26s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a6bae0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5w2hj7t2fgmv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "name": "fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:32.768656716Z",
      "endTime": "2026-09-28T13:22:40.579367905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m26s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0xc000ea49c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-q4hdh7csw5qs" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508",
      "name": "daab790a-4984-45ab-a2a3-a3f37cd79508",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:48.885600114Z",
      "endTime": "2026-09-28T13:22:50.894287651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m10s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0xc000ea49c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-q4hdh7csw5qs" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508",
      "name": "daab790a-4984-45ab-a2a3-a3f37cd79508",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:48.885600114Z",
      "endTime": "2026-09-28T13:22:50.894287651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m10s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d60180>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-txbk9bmdh8wg" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "name": "7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:22.508354202Z",
      "endTime": "2026-09-28T13:22:30.229773982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 26m20s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d60180>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-txbk9bmdh8wg" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "name": "7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:22.508354202Z",
      "endTime": "2026-09-28T13:22:30.229773982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 26m20s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2s722b
Unexpected error:
    <*fmt.wrapError | 0xc000bfeb60>: 
    failed to create HCP cluster np-noedge-2s722b: failed waiting for cluster="np-noedge-2s722b" in resourcegroup="rg-np-noedge-2s722b-ttwnzxstsj4p" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93",
      "name": "18b3752b-dfe7-413c-83e0-e4570361ec93",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:09.735885483Z",
      "endTime": "2026-09-28T05:23:14.561838226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2s722b
Unexpected error:
    <*fmt.wrapError | 0xc000bfeb60>: 
    failed to create HCP cluster np-noedge-2s722b: failed waiting for cluster="np-noedge-2s722b" in resourcegroup="rg-np-noedge-2s722b-ttwnzxstsj4p" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93",
      "name": "18b3752b-dfe7-413c-83e0-e4570361ec93",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:09.735885483Z",
      "endTime": "2026-09-28T05:23:14.561838226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000fc65a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-6vnwh8vjnz8r" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f",
      "name": "0b3163de-a897-4f45-9b77-e785bdfab09f",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.798033495Z",
      "endTime": "2026-09-28T05:22:54.610455796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000fc65a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-6vnwh8vjnz8r" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f",
      "name": "0b3163de-a897-4f45-9b77-e785bdfab09f",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.798033495Z",
      "endTime": "2026-09-28T05:22:54.610455796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8fvg54" with version 4.22.15 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ae0680>: 
    failed to create HCP cluster cluster-zstream-4-22-8fvg54: failed waiting for cluster="cluster-zstream-4-22-8fvg54" in resourcegroup="rg-zstream-upgrade-4-22-8zfwm76xgnlj" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0",
      "name": "847235cf-8889-474c-8fc4-418de69191f0",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:18.057920805Z",
      "endTime": "2026-09-28T19:39:25.918712924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8fvg54" with version 4.22.15 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ae0680>: 
    failed to create HCP cluster cluster-zstream-4-22-8fvg54: failed waiting for cluster="cluster-zstream-4-22-8fvg54" in resourcegroup="rg-zstream-upgrade-4-22-8zfwm76xgnlj" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0",
      "name": "847235cf-8889-474c-8fc4-418de69191f0",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:18.057920805Z",
      "endTime": "2026-09-28T19:39:25.918712924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-27p526" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d20a60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-27p526: failed waiting for cluster="cp-ystream-upgrade-4-22-27p526" in resourcegroup="rg-cp-ystream-upgrade-4-22-7drm72sqb8zr" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "name": "3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:49.293974095Z",
      "endTime": "2026-09-28T19:39:55.897408712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-27p526" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d20a60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-27p526: failed waiting for cluster="cp-ystream-upgrade-4-22-27p526" in resourcegroup="rg-cp-ystream-upgrade-4-22-7drm72sqb8zr" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "name": "3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:49.293974095Z",
      "endTime": "2026-09-28T19:39:55.897408712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000ce94a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j9cxlp" in resourcegroup="rg-np-noedge-j9cxlp-9zvfnm962trj" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "name": "3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "status": "Failed",
      "startTime": "2026-09-28T17:41:00.477362299Z",
      "endTime": "2026-09-28T18:00:09.600349173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000ce94a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j9cxlp" in resourcegroup="rg-np-noedge-j9cxlp-9zvfnm962trj" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "name": "3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "status": "Failed",
      "startTime": "2026-09-28T17:41:00.477362299Z",
      "endTime": "2026-09-28T18:00:09.600349173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z"
Unexpected error:
    <*fmt.wrapError | 0xc000f712c0>: 
    failed to create NodePool np-wsvb2z: failed waiting for nodepool="np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z" in resourcegroup="rg-candidate-4-20-slrjsf69bdm7" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7",
      "name": "119c8117-1a73-44d6-8ce8-38b6990193b7",
      "status": "Failed",
      "startTime": "2026-09-28T17:43:51.079865069Z",
      "endTime": "2026-09-28T18:02:59.523800413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z"
Unexpected error:
    <*fmt.wrapError | 0xc000f712c0>: 
    failed to create NodePool np-wsvb2z: failed waiting for nodepool="np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z" in resourcegroup="rg-candidate-4-20-slrjsf69bdm7" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7",
      "name": "119c8117-1a73-44d6-8ce8-38b6990193b7",
      "status": "Failed",
      "startTime": "2026-09-28T17:43:51.079865069Z",
      "endTime": "2026-09-28T18:02:59.523800413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71242104613528387194880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal: Timed out after <duration>s.
full failure pattern:
Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal: Timed out after <duration>s.
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:164 with:
expected HTTP 200 from sample app via internal LB, got 503
Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:164 with:
expected HTTP 200 from sample app via internal LB, got 503
Expected
    <string>: ...
to equal
    <string>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-apiserver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-26r6nt" with version 4.20.39 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106e580>: 
    failed to create HCP cluster cluster-zstream-4-20-26r6nt: failed waiting for cluster="cluster-zstream-4-20-26r6nt" in resourcegroup="rg-zstream-upgrade-4-20-6bpd6fpsn6s4" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "name": "4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:43.832400833Z",
      "endTime": "2026-09-28T13:22:51.48967289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-apiserver, openshift-route-controller-manager, catalog-operator, community-operators-catalog, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-storage-operator, openshift-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, ingress-operator, cluster-policy-controller, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-26r6nt" with version 4.20.39 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106e580>: 
    failed to create HCP cluster cluster-zstream-4-20-26r6nt: failed waiting for cluster="cluster-zstream-4-20-26r6nt" in resourcegroup="rg-zstream-upgrade-4-20-6bpd6fpsn6s4" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "name": "4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:43.832400833Z",
      "endTime": "2026-09-28T13:22:51.48967289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-apiserver, openshift-route-controller-manager, catalog-operator, community-operators-catalog, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-storage-operator, openshift-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, ingress-operator, cluster-policy-controller, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc0010c41e0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-jfwb2dgmj26t" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "name": "a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:17.858913168Z",
      "endTime": "2026-09-28T05:23:24.118546984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc0010c41e0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-jfwb2dgmj26t" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "name": "a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:17.858913168Z",
      "endTime": "2026-09-28T05:23:24.118546984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, kube-apiserver, openshift-apiserver, packageserver, redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010432a0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-66ncz7vqwm2z" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86",
      "name": "c35d648d-395d-429f-9f2d-24eed4314d86",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:11.81451702Z",
      "endTime": "2026-09-28T19:39:16.076223121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, community-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010432a0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-66ncz7vqwm2z" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86",
      "name": "c35d648d-395d-429f-9f2d-24eed4314d86",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:11.81451702Z",
      "endTime": "2026-09-28T19:39:16.076223121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, community-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6kj9nw5mjrc2/cluster-hs-pre-fvbpxs should provision
Unexpected error:
    <*fmt.wrapError | 0xc000db85e0>: 
    failed to create HCP cluster cluster-hs-pre-fvbpxs: failed waiting for cluster="cluster-hs-pre-fvbpxs" in resourcegroup="rg-hs-pre--6kj9nw5mjrc2" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd",
      "name": "2f78ab67-468f-449c-890b-25423489ffcd",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.674200437Z",
      "endTime": "2026-09-28T05:23:04.406727689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6kj9nw5mjrc2/cluster-hs-pre-fvbpxs should provision
Unexpected error:
    <*fmt.wrapError | 0xc000db85e0>: 
    failed to create HCP cluster cluster-hs-pre-fvbpxs: failed waiting for cluster="cluster-hs-pre-fvbpxs" in resourcegroup="rg-hs-pre--6kj9nw5mjrc2" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd",
      "name": "2f78ab67-468f-449c-890b-25423489ffcd",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.674200437Z",
      "endTime": "2026-09-28T05:23:04.406727689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ae31a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-5cbtfzkp497x" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "name": "6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:33.784163345Z",
      "endTime": "2026-09-28T05:22:34.354425424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ae31a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-5cbtfzkp497x" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "name": "6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:33.784163345Z",
      "endTime": "2026-09-28T05:22:34.354425424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cb0840>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rbwvmrnzw8j5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e",
      "name": "5bdae3db-e612-4f27-8693-02200538e08e",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:23.74994885Z",
      "endTime": "2026-09-28T05:23:24.026384001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cb0840>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rbwvmrnzw8j5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e",
      "name": "5bdae3db-e612-4f27-8693-02200538e08e",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:23.74994885Z",
      "endTime": "2026-09-28T05:23:24.026384001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00106c320>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-qdqkz9rfmldx" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23",
      "name": "0c723202-68f5-41c7-a5d4-762157d31d23",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.997312592Z",
      "endTime": "2026-09-28T05:22:54.143428969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00106c320>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-qdqkz9rfmldx" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23",
      "name": "0c723202-68f5-41c7-a5d4-762157d31d23",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.997312592Z",
      "endTime": "2026-09-28T05:22:54.143428969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gng5crkc9wh8/cluster-candidate-4-22-w8wr72 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00117a200>: 
    failed to create HCP cluster cluster-candidate-4-22-w8wr72: failed waiting for cluster="cluster-candidate-4-22-w8wr72" in resourcegroup="rg-candidate-4-22-gng5crkc9wh8" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "name": "00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.970731237Z",
      "endTime": "2026-09-28T05:23:04.188577598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, router; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gng5crkc9wh8/cluster-candidate-4-22-w8wr72 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00117a200>: 
    failed to create HCP cluster cluster-candidate-4-22-w8wr72: failed waiting for cluster="cluster-candidate-4-22-w8wr72" in resourcegroup="rg-candidate-4-22-gng5crkc9wh8" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "name": "00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.970731237Z",
      "endTime": "2026-09-28T05:23:04.188577598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, router; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint <hcp-api-host> failed: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-rxbdcb" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001090b40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-rxbdcb: failed waiting for cluster="cp-ystream-upgrade-4-23-rxbdcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-jvcp8dmnbrt7" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "name": "1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:56.829077997Z",
      "endTime": "2026-09-28T07:47:03.158495671Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud failed: Get \"https://api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud:443/healthz?verbose\": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-rxbdcb" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001090b40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-rxbdcb: failed waiting for cluster="cp-ystream-upgrade-4-23-rxbdcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-jvcp8dmnbrt7" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "name": "1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:56.829077997Z",
      "endTime": "2026-09-28T07:47:03.158495671Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud failed: Get \"https://api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud:443/healthz?verbose\": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104457246334259200
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f0f2c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p82fzl2z5vk5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022",
      "name": "03f41a1e-bd33-4df1-9f47-56824bb74022",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:14.362679822Z",
      "endTime": "2026-09-28T05:24:15.510395217Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f0f2c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p82fzl2z5vk5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022",
      "name": "03f41a1e-bd33-4df1-9f47-56824bb74022",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:14.362679822Z",
      "endTime": "2026-09-28T05:24:15.510395217Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, cluster-image-registry-operator, community-operators-catalog, hosted-cluster-config-operator, kube-apiserver, openshift-apiserver, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-c89c6r" with version 5.0.0-rc.3 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b10480>: 
    failed to create HCP cluster cluster-zstream-5-0-c89c6r: failed waiting for cluster="cluster-zstream-5-0-c89c6r" in resourcegroup="rg-zstream-upgrade-5-0-6kc9p4vcvjqq" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "name": "37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:45.206355596Z",
      "endTime": "2026-09-28T13:22:51.172490782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 26m23s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-c89c6r" with version 5.0.0-rc.3 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b10480>: 
    failed to create HCP cluster cluster-zstream-5-0-c89c6r: failed waiting for cluster="cluster-zstream-5-0-c89c6r" in resourcegroup="rg-zstream-upgrade-5-0-6kc9p4vcvjqq" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "name": "37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:45.206355596Z",
      "endTime": "2026-09-28T13:22:51.172490782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 26m23s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, kube-apiserver, redhat-marketplace-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bf61a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lhz84lf5nw65" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119",
      "name": "045ddd50-d43d-42a0-bcda-90888c378119",
      "status": "Failed",
      "startTime": "2026-09-28T19:11:03.945176993Z",
      "endTime": "2026-09-28T19:40:05.721943753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m29s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bf61a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lhz84lf5nw65" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119",
      "name": "045ddd50-d43d-42a0-bcda-90888c378119",
      "status": "Failed",
      "startTime": "2026-09-28T19:11:03.945176993Z",
      "endTime": "2026-09-28T19:40:05.721943753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m29s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, kube-controller-manager, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6mht66l8hhvp/cluster-candidate-4-22-hxvn25 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00070f360>: 
    failed to create HCP cluster cluster-candidate-4-22-hxvn25: failed waiting for cluster="cluster-candidate-4-22-hxvn25" in resourcegroup="rg-candidate-4-22-6mht66l8hhvp" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "name": "889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:32.906905274Z",
      "endTime": "2026-09-28T19:39:35.528660951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6mht66l8hhvp/cluster-candidate-4-22-hxvn25 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00070f360>: 
    failed to create HCP cluster cluster-candidate-4-22-hxvn25: failed waiting for cluster="cluster-candidate-4-22-hxvn25" in resourcegroup="rg-candidate-4-22-6mht66l8hhvp" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "name": "889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:32.906905274Z",
      "endTime": "2026-09-28T19:39:35.528660951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc00060c440>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-br7vgttbd9zx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "name": "15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:44.986989982Z",
      "endTime": "2026-09-28T19:39:45.910063992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 24m39s ago; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc00060c440>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-br7vgttbd9zx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "name": "15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:44.986989982Z",
      "endTime": "2026-09-28T19:39:45.910063992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 24m39s ago; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailab...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, openshift-apiserver, packageserver, redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4rl6n98blqls/cluster-candidate-5-0-9zqn4p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f22920>: 
    failed to create HCP cluster cluster-candidate-5-0-9zqn4p: failed waiting for cluster="cluster-candidate-5-0-9zqn4p" in resourcegroup="rg-candidate-5-0-4rl6n98blqls" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927",
      "name": "b919a89b-ddbd-409e-ae0b-825760357927",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:07.120894243Z",
      "endTime": "2026-09-28T19:39:16.873419842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4rl6n98blqls/cluster-candidate-5-0-9zqn4p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f22920>: 
    failed to create HCP cluster cluster-candidate-5-0-9zqn4p: failed waiting for cluster="cluster-candidate-5-0-9zqn4p" in resourcegroup="rg-candidate-5-0-4rl6n98blqls" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927",
      "name": "b919a89b-ddbd-409e-ae0b-825760357927",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:07.120894243Z",
      "endTime": "2026-09-28T19:39:16.873419842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroup...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/<uuid>' at line '<line>' and column '<column>'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'; provider Microsoft.Authorization
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ca2e60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-z4h7jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-z4h7jf/operationStatuses/08584110018702456355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name/providers/Microsoft.Authorization/roleAssignments/ce4c794c-50e9-565e-99cc-b7dd7e08c5a9' at line '1' and column '29054'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 29054,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/e5697265-1ef6-54a3-8e48-b96178ae0611' at line '1' and column '30683'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 30683,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ca2e60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-z4h7jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-z4h7jf/operationStatuses/08584110018702456355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name/providers/Microsoft.Authorization/roleAssignments/ce4c794c-50e9-565e-99cc-b7dd7e08c5a9' at line '1' and column '29054'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 29054,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/e5697265-1ef6-54a3-8e48-b96178ae0611' at line '1' and column '30683'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 30683,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71042104562485683032064
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRUpdateHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc002725230>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w8pbb4pwwmm7" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-w8pbb4pwwmm7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc002725230>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w8pbb4pwwmm7" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-w8pbb4pwwmm7, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet <namespace>/global-pull-secret-syncer has no desired pods scheduled
full failure pattern:
VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet <namespace>/global-pull-secret-syncer has no desired pods scheduled
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:183]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0xc000fb0020>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet kube-system/global-pull-secret-syncer has no desired pods scheduled
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:183]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0xc000fb0020>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet kube-system/global-pull-secret-syncer has no desired pods scheduled
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71242104613528387194880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-...
full failure pattern:
VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullnsjbs8pwqtmk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00101eb60>: 
    VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e8a1ddc1-f3e1-4c99-ae14-bbe272e0b702; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d311f24-e917-4750-a06e-94b359669923
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullnsjbs8pwqtmk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00101eb60>: 
    VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e8a1ddc1-f3e1-4c99-ae14-bbe272e0b702; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d311f24-e917-4750-a06e-94b359669923
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71812104634571122282496
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
full failure pattern:
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001664240>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001664240>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72312104701088555864064
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] KubeNodeUnreachable fired
full failure pattern:
alert [svc] KubeNodeUnreachable fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:21:03Z
  Ended:       2026-09-28T13:27:01Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="prometheus", node="aks-userswft3-30800232-vmss000000", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft3-30800232-vmss000000 is unreachable and some workloads may be rescheduled.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeNodeUnreachable does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
full failure pattern:
failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.88%1 of 114 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-b6k7qp"
Unexpected error:
    <*fmt.wrapError | 0xc000722f40>: 
    failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-b6k7qp"
Unexpected error:
    <*fmt.wrapError | 0xc000722f40>: 
    failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
alert [svc] FrontendPathMedianLatency fired
full failure pattern:
alert [svc] FrontendPathMedianLatency fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert4640.35%46 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 22: 21 · Sep 23: 22 · Sep 24: 24 · Sep 25: 30 · Sep 26: 5 · Sep 27: 6 · Sep 28: 46none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T20:22:39Z
  Ended:       2026-09-28T20:28:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9312256-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9312256-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:18:56Z
  Ended:       2026-09-28T19:51:55Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0189952-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0189952-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T18:59:41Z
  Ended:       2026-09-28T19:05:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3028352-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3028352-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire46
Affected runs (46)
Date (UTC)Associated PRProw job
PR #69222104648687849312256
PR #71782104639037800189952
PR #70532104626712733028352
PR #69682104623441356263424
PR #72272104620444073070592
PR #71042104605657771544576
PR #70202104605132682432512
PR #72242104593036766875648
PR #71702104592047892598784
PR #70682104592038186979328
PR #71422104591950119178240
PR #69222104588552477085696
PR #72142104587545261117440
PR #71432104586688620662784
PR #72022104576830420815872
PR #69002104576709901684736
PR #72252104564648333283328
PR #70542104561257146224640
PR #72212104551522074365952
PR #69222104551505146155008
PR #71782104539007848288256
PR #71822104529106451304448
PR #72172104526339443789824
PR #72182104519303637241856
PR #71572104500979540955136
PR #68212104500972830068736
PR #72022104500902567088128
PR #72032104500868232515584
PR #69002104485944341041152
PR #62262104484432940371968
PR #71582104483794588274688
PR #72162104472130388234240
PR #72142104466453632651264
PR #71782104457246334259200
PR #72022104441054601154560
PR #72032104441035131195392
PR #68212104440974754189312
PR #70682104440930432978944
PR #68692104433568867422208
PR #62262104429033809776640
PR #71782104421180684374016
PR #71782104380048239759360
PR #70682104364976289353728
PR #68212104364960745263104
PR #71572104364925383086080
PR #72032104363449923407872
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert76.14%7 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 22: 6 · Sep 23: 4 · Sep 24: 2 · Sep 25: 12 · Sep 26: 0 · Sep 27: 2 · Sep 28: 7none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T00:14:18Z
  Ended:       2026-09-29T00:26:21Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4504064-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:20:58Z
  Ended:       2026-09-28T16:27:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0662784-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T12:00:53Z
  Ended:       2026-09-28T12:14:51Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0514944-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T12:06:51Z
  Ended:       2026-09-28T12:18:52Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0514944-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #71242104709623964504064
PR #71432104586688620662784
PR #68842104529166530514944
PR #71822104529106451304448
PR #70682104500977863233536
PR #72022104500902567088128
PR #71572104364925383086080
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e76.14%7 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 22: 2 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 4 · Sep 27: 4 · Sep 28: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #69222104551505146155008
PR #72032104500868232515584
PR #71692104492976943140864
PR #68212104440974754189312
PR #71852104381379776090112
PR #71782104380048239759360
PR #72022104362505772994560
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
alert54.39%5 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 22: 3 · Sep 23: 0 · Sep 24: 0 · Sep 25: 2 · Sep 26: 1 · Sep 27: 0 · Sep 28: 5none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T21:47:52Z
  Ended:       2026-09-28T22:02:50Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:12:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z8g6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:15:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:15:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:16:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:17:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z8g6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #72292104669678952517632
PR #71782104639037800189952
PR #69002104485944341041152
PR #71782104457246334259200
PR #71782104421180684374016
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert54.39%5 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 22: 3 · Sep 23: 2 · Sep 24: 4 · Sep 25: 1 · Sep 26: 2 · Sep 27: 1 · Sep 28: 5none
Full failure examples (3)
alert fired 79 time(s) (63 unknown, 16 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:22:59Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="router-5b97c8c479-td767", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/router-5b97c8c479-td767 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T19:23:58Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rbkaemrkauofk92ubess3a0bmc8a-v20261001", pod="router-688fdf9d74-7b7qb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rbkaemrkauofk92ubess3a0bmc8a-v20261001/router-688fdf9d74-7b7qb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T19:25:04Z
  Ended:       2026-09-28T19:47:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster", pod="router-7856497d5-p9s72", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster/router-7856497d5-p9s72 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T19:25:04Z
  Ended:       2026-09-28T19:47:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster", pod="router-7856497d5-ptl86", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster/router-7856497d5-ptl86 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="ovnkube-control-plane-5fd96b4979-gbmgt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/ovnkube-control-plane-5fd96b4979-gbmgt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="ingress-operator-7d88f7b5dc-drxfd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/ingress-operator-7d88f7b5dc-drxfd has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-file-csi-driver-controller-854b478df8-v98lf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-file-csi-driver-controller-854b478df8-v98lf has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-disk-csi-driver-controller-5785d6645d-6xw4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-disk-csi-driver-controller-5785d6645d-6xw4f has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-disk-csi-driver-controller-5c84fd46c4-mpzts", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-disk-csi-driver-controller-5c84fd46c4-mpzts has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-file-csi-driver-controller-559d66d9f8-v468g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-file-csi-driver-controller-559d66d9f8-v468g has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-file-csi-driver-controller-9688c9fcd-vkkwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-file-csi-driver-controller-9688c9fcd-vkkwc has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-disk-csi-driver-controller-df8985bcf-xr5pq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-disk-csi-driver-controller-df8985bcf-xr5pq has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-file-csi-driver-controller-6f4b9c57cd-gpkcq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-file-csi-driver-controller-6f4b9c57cd-gpkcq has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T19:29:02Z
  Ended:       2026-09-28T19:35:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6qhsdqg59jb18mtn75mdepd994t-p1e0l2c6g7e7x9p", pod="azure-disk-csi-driver-controller-d5b4b46ff-swrjk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6qhsdqg59jb18mtn75mdepd994t-p1e0l2c6g7e7x9p/azure-disk-csi-driver-controller-d5b4b46ff-swrjk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T19:29:02Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="ignition-server-6cf4c4cdbb-dcl2n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/ignition-server-6cf4c4cdbb-dcl2n has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="cluster-storage-operator-665d6949b6-6fw7x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/cluster-storage-operator-665d6949b6-6fw7x has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:37:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="ingress-operator-fb65c85-29jmc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/ingress-operator-fb65c85-29jmc has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="dns-operator-bfc958595-f2lvq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/dns-operator-bfc958595-f2lvq has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="kube-controller-manager-847bbfc79-gdtt6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/kube-controller-manager-847bbfc79-gdtt6 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="cluster-network-operator-6995f687db-6cfj4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/cluster-network-operator-6995f687db-6cfj4 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-28T19:33:01Z
  Ended:       2026-09-28T19:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rb6d2v8ktanhkbba2l3aora65d7u-t1b3k0j9k6s4g3e", pod="kube-apiserver-5f69cc7c69-66mrs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rb6d2v8ktanhkbba2l3aora65d7u-t1b3k0j9k6s4g3e/kube-apiserver-5f69cc7c69-66mrs has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-network-operator-79b745b7-2lm4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-network-operator-79b745b7-2lm4f has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="kube-apiserver-c7844b576-98nm5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/kube-apiserver-c7844b576-98nm5 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-version-operator-767f75f6bd-czn9v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-version-operator-767f75f6bd-czn9v has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="network-node-identity-688597d98d-5nwfj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/network-node-identity-688597d98d-5nwfj has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="kube-apiserver-674ffb8fcf-w2vnq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/kube-apiserver-674ffb8fcf-w2vnq has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="csi-snapshot-controller-78d9d968f6-lktl7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/csi-snapshot-controller-78d9d968f6-lktl7 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="kube-apiserver-7f4977d86c-ggtv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/kube-apiserver-7f4977d86c-ggtv4 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="community-operators-catalog-77957f8659-vrvjj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/community-operators-catalog-77957f8659-vrvjj has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="multus-admission-controller-6659c5d88f-hgsf6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/multus-admission-controller-6659c5d88f-hgsf6 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="router-7964b968d8-s9n7t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/router-7964b968d8-s9n7t has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="ignition-server-6cf4c4cdbb-xqn6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/ignition-server-6cf4c4cdbb-xqn6v has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="openshift-apiserver-79bff85c55-rmr6j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/openshift-apiserver-79bff85c55-rmr6j has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="openshift-apiserver-f4f5cc6bf-j27d5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/openshift-apiserver-f4f5cc6bf-j27d5 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r619d7598ajep6f8mtkq1cmmun0r-g6f2d0d9e8y9l8w", pod="openshift-apiserver-855bf867d4-5dj69", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r619d7598ajep6f8mtkq1cmmun0r-g6f2d0d9e8y9l8w/openshift-apiserver-855bf867d4-5dj69 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="openshift-apiserver-548d95f77-j9rt4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/openshift-apiserver-548d95f77-j9rt4 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="kube-apiserver-8488cbccb8-9cntq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/kube-apiserver-8488cbccb8-9cntq has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="openshift-apiserver-59cf74f469-2pkbk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/openshift-apiserver-59cf74f469-2pkbk has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-image-registry-operator-5f9967bf67-m75d8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-image-registry-operator-5f9967bf67-m75d8 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="router-6966cb469f-jdx2g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/router-6966cb469f-jdx2g has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="kube-apiserver-6474ccdcd7-j5pkn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/kube-apiserver-6474ccdcd7-j5pkn has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="kube-apiserver-7877d486f4-dwd29", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/kube-apiserver-7877d486f4-dwd29 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="network-node-identity-79f47f8db7-c6lsb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/network-node-identity-79f47f8db7-c6lsb has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="network-node-identity-6bf46b86d-r6lxm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/network-node-identity-6bf46b86d-r6lxm has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:48:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="router-d5fd8c684-npjp8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/router-d5fd8c684-npjp8 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="openshift-apiserver-7fccc5c996-2fqjh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/openshift-apiserver-7fccc5c996-2fqjh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="certified-operators-catalog-56fcb687d5-x55km", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/certified-operators-catalog-56fcb687d5-x55km has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="openshift-apiserver-58689dc575-d9psz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/openshift-apiserver-58689dc575-d9psz has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="openshift-apiserver-fb96d7cd5-n8vx8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/openshift-apiserver-fb96d7cd5-n8vx8 has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="kube-apiserver-58866c8f4c-rssbm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/kube-apiserver-58866c8f4c-rssbm has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="openshift-apiserver-57597ccd79-kdsdf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/openshift-apiserver-57597ccd79-kdsdf has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="network-node-identity-6b45dfb8b5-gpsw6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/network-node-identity-6b45dfb8b5-gpsw6 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="router-5b97c8c479-87d9j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/router-5b97c8c479-87d9j has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="network-node-identity-cf9fb8b88-4lqsp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/network-node-identity-cf9fb8b88-4lqsp has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="kube-apiserver-6776ddb8b4-jtcfw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/kube-apiserver-6776ddb8b4-jtcfw has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-09-28T19:36:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-file-csi-driver-controller-68d9cdbd6c-bjqhs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-file-csi-driver-controller-68d9cdbd6c-bjqhs has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-disk-csi-driver-controller-688579f7b8-gcwdq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-disk-csi-driver-controller-688579f7b8-gcwdq has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-disk-csi-driver-controller-778588d6c6-grn9w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-disk-csi-driver-controller-778588d6c6-grn9w has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="kube-controller-manager-5488d69fb6-p9ltk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/kube-controller-manager-5488d69fb6-p9ltk has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="kube-controller-manager-9864948dd-qzgnt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/kube-controller-manager-9864948dd-qzgnt has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="ovnkube-control-plane-58c8f9b6c5-45j6m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/ovnkube-control-plane-58c8f9b6c5-45j6m has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="network-node-identity-789b89cf4b-2v5sk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/network-node-identity-789b89cf4b-2v5sk has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-09-28T19:38:02Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re3hoaml91vd90m68oiaekbg44as-e7l1h2p9d0o6f4j", pod="openshift-apiserver-5f8cc55fd4-bm995", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re3hoaml91vd90m68oiaekbg44as-e7l1h2p9d0o6f4j/openshift-apiserver-5f8cc55fd4-bm995 has been in a non-ready state for longer than 5 minutes.
alert fired 39 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qwlf6", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qwlf6 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-w28wh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-w28wh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-zxx8r", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-zxx8r has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-g76qj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-g76qj has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-d9qzz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-d9qzz has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-fpkv2", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-fpkv2 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-cwzx8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-cwzx8 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qm6c8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qm6c8 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-4996z", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-4996z has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-675f5", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-675f5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-gd9k9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-gd9k9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-z58gp", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-z58gp has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-4j2hv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-4j2hv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-8n4fj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-8n4fj has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-8q8gs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-8q8gs has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-ddxpz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-ddxpz has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-9kqwf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-9kqwf has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-tpk28", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-tpk28 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-wpwxf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-wpwxf has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-c9pkv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-c9pkv has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-r2z8h", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-r2z8h has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-v6lst", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-v6lst has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-6nc22", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-6nc22 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-52zbx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-52zbx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-j8nbh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-j8nbh has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-f9lhx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-f9lhx has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dp559", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dp559 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-j4pwc", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-j4pwc has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-p9w7c", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-p9w7c has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-hqdp9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-hqdp9 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dgb65", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dgb65 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-rvk8h", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-rvk8h has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-mpfhk", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-mpfhk has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qm2ts", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qm2ts has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-27cpw", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-27cpw has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-mxsvz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-mxsvz has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-hgtjs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-hgtjs has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-c8pzf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-c8pzf has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dvzkd", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dvzkd has been in a non-ready state for longer than 5 minutes.
alert fired 54 time(s) (38 unknown, 16 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:01Z
  Ended:       2026-09-28T13:29:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41bfp4l0n502nn624hggcdkif3fakm-l7r5c9r0t4i6i9i", pod="router-5f8dd5cdc8-2x7hv", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41bfp4l0n502nn624hggcdkif3fakm-l7r5c9r0t4i6i9i/router-5f8dd5cdc8-2x7hv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:13:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="cluster-network-operator-745b94c8d6-xgbrm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/cluster-network-operator-745b94c8d6-xgbrm has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:29:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster", pod="router-7f5958d584-h7qql", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster/router-7f5958d584-h7qql has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="openshift-apiserver-77dbbb84d4-g84qr", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/openshift-apiserver-77dbbb84d4-g84qr has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41aorgo7k4e5t9d3gl5386u8qrr1ok-b1g2t3h0x4w9w8x", pod="router-869d798bbd-2c6kt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41aorgo7k4e5t9d3gl5386u8qrr1ok-b1g2t3h0x4w9w8x/router-869d798bbd-2c6kt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:37:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster", pod="router-7f5958d584-xsp4j", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster/router-7f5958d584-xsp4j has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T13:08:02Z
  Ended:       2026-09-28T13:35:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41bju2fkvns9a7dk4sqfvummalvm0f-t0e8p3i4e9p7l9u", pod="router-6449799f6-vwm42", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41bju2fkvns9a7dk4sqfvummalvm0f-t0e8p3i4e9p7l9u/router-6449799f6-vwm42 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T13:08:02Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="ignition-server-proxy-67ffb7567d-ndltn", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/ignition-server-proxy-67ffb7567d-ndltn has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T13:08:03Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41an05susenrnl0dfs40rcas06ch4c-f7z5i4t9g6o0n5p", pod="ignition-server-proxy-8699545598-fvwm5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41an05susenrnl0dfs40rcas06ch4c-f7z5i4t9g6o0n5p/ignition-server-proxy-8699545598-fvwm5 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T13:08:03Z
  Ended:       2026-09-28T13:26:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-controller-manager-867d6bcbc8-7zqfz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-controller-manager-867d6bcbc8-7zqfz has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T13:16:02Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="openshift-apiserver-db5f58d87-n2wk7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/openshift-apiserver-db5f58d87-n2wk7 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="kube-apiserver-85d5846f96-gpf28", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/kube-apiserver-85d5846f96-gpf28 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", pod="openshift-apiserver-85677f65c4-wrvhg", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics/openshift-apiserver-85677f65c4-wrvhg has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="network-node-identity-7549b9758b-4wb6s", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/network-node-identity-7549b9758b-4wb6s has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="openshift-apiserver-77dbbb84d4-29r9g", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/openshift-apiserver-77dbbb84d4-29r9g has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-apiserver-5c98b4447b-wvxv8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-apiserver-5c98b4447b-wvxv8 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="kube-apiserver-745ff9b4db-srgc7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/kube-apiserver-745ff9b4db-srgc7 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="openshift-apiserver-84fcf4bc4d-mcmsx", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/openshift-apiserver-84fcf4bc4d-mcmsx has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="openshift-apiserver-b6b777476-w4j8h", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/openshift-apiserver-b6b777476-w4j8h has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="network-node-identity-57484c8884-trf2c", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/network-node-identity-57484c8884-trf2c has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="openshift-apiserver-5c68654d4b-dbjjt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/openshift-apiserver-5c68654d4b-dbjjt has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="openshift-apiserver-68c54f7686-xjxd2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/openshift-apiserver-68c54f7686-xjxd2 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="kube-apiserver-86c59c88cb-qrxm8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/kube-apiserver-86c59c88cb-qrxm8 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-controller-manager-867d6bcbc8-bpsb5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-controller-manager-867d6bcbc8-bpsb5 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="network-node-identity-65ff54759c-92pcp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/network-node-identity-65ff54759c-92pcp has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="network-node-identity-594c4dccf4-lm2n6", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/network-node-identity-594c4dccf4-lm2n6 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", pod="kube-apiserver-b8d4f754f-dqwb6", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics/kube-apiserver-b8d4f754f-dqwb6 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="kube-apiserver-957c77ffc-rg8wb", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/kube-apiserver-957c77ffc-rg8wb has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="kube-apiserver-5c9ff4f96c-n5vjw", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/kube-apiserver-5c9ff4f96c-n5vjw has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="openshift-apiserver-554784946b-mjxzz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/openshift-apiserver-554784946b-mjxzz has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="kube-apiserver-65f96cf6cc-d9g85", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/kube-apiserver-65f96cf6cc-d9g85 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-28T13:19:08Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anli60mk7obvb9qscekiln1r7nlh-z0b5k9r4h8u4l9q", pod="openshift-apiserver-5f4c5b7f7c-l5gpc", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anli60mk7obvb9qscekiln1r7nlh-z0b5k9r4h8u4l9q/openshift-apiserver-5f4c5b7f7c-l5gpc has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-28T13:20:08Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="multus-admission-controller-78665bcc7c-5f2bh", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/multus-admission-controller-78665bcc7c-5f2bh has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="csi-snapshot-controller-operator-74759598d-8dhlz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/csi-snapshot-controller-operator-74759598d-8dhlz has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="cluster-storage-operator-55cc4d6675-r7fxd", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/cluster-storage-operator-55cc4d6675-r7fxd has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="cluster-network-operator-5dc56d94dc-rmrjd", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/cluster-network-operator-5dc56d94dc-rmrjd has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="dns-operator-6d45bd4dc-c5h9t", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/dns-operator-6d45bd4dc-c5h9t has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="ingress-operator-6c4557f7b9-x9crg", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/ingress-operator-6c4557f7b9-x9crg has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71432104586688620662784
PR #71782104539007848288256
PR #71782104457246334259200
PR #71782104421180684374016
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e21.75%2 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 22: 91 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 14none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00172e090>: 
    resource group "complex-cilium-kv-k9knbpmhbg7v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k9knbpmhbg7v" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "name": "4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:43.521207454Z",
      "endTime": "2026-09-28T09:54:45.223507829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u15umjtvbfn7or4kh25ftp6mt8ubm still exists (deletion dispatched at 2026-09-28T09:30:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00172e090>: 
    resource group "complex-cilium-kv-k9knbpmhbg7v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k9knbpmhbg7v" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "name": "4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:43.521207454Z",
      "endTime": "2026-09-28T09:54:45.223507829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u15umjtvbfn7or4kh25ftp6mt8ubm still exists (deletion dispatched at 2026-09-28T09:30:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002958120>: 
    resource group "rg-candidate-4-20-n6x2rfcpmr5t": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bkc8nt" in resourcegroup="rg-candidate-4-20-n6x2rfcpmr5t" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "name": "4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:44.717303002Z",
      "endTime": "2026-09-28T09:54:45.21106923Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ce2mv9aje3edhhstj5ffjhl0mpi still exists (deletion dispatched at 2026-09-28T09:31:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002958120>: 
    resource group "rg-candidate-4-20-n6x2rfcpmr5t": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bkc8nt" in resourcegroup="rg-candidate-4-20-n6x2rfcpmr5t" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "name": "4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:44.717303002Z",
      "endTime": "2026-09-28T09:54:45.21106923Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ce2mv9aje3edhhstj5ffjhl0mpi still exists (deletion dispatched at 2026-09-28T09:31:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001010288>: 
    resource group "clusternp128-cr6m72hbhthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cr6m72hbhthl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "name": "2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "status": "Failed",
      "startTime": "2026-09-28T09:26:14.153004619Z",
      "endTime": "2026-09-28T09:50:15.107483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ep16n5al4fc1ggfac6imb350l still exists (deletion dispatched at 2026-09-28T09:26:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001010288>: 
    resource group "clusternp128-cr6m72hbhthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cr6m72hbhthl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "name": "2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "status": "Failed",
      "startTime": "2026-09-28T09:26:14.153004619Z",
      "endTime": "2026-09-28T09:50:15.107483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ep16n5al4fc1ggfac6imb350l still exists (deletion dispatched at 2026-09-28T09:26:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (14)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69002104485944341041152
PR #70412104360746694479872
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert21.75%2 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 22: 2 · Sep 23: 2 · Sep 24: 1 · Sep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:45:04Z
  Ended:       2026-09-28T19:51:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="klusterlet-2t46re8cgt25tnc3e3r05ha4ola9p2m6", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t46re8cgt25tnc3e3r05ha4ola9p2m6/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/services-j4374016", namespace="klusterlet-2t3qakt150613r65up3s1cocvavjoksi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t3qakt150613r65up3s1cocvavjoksi/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/services-j4374016", namespace="klusterlet-2t3qakt150613r65up3s1cocvavjoksi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t3qakt150613r65up3s1cocvavjoksi/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71782104421180684374016
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e21.75%2 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 22: 1 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002737518>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc002737518>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullhctmh8kwn4gw.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00011dc00>: 
    VerifyImagePulled(acrpullhctmh8kwn4gw.azurecr.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullhctmh8kwn4gw.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00011dc00>: 
    VerifyImagePulled(acrpullhctmh8kwn4gw.azurecr.io/ubi-minimal) timed out after 3m0.008s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71812104634571122282496
PR #71812104578384683077632
alert [svc] KubeDaemonSetRolloutStuck fired
full failure pattern:
alert [svc] KubeDaemonSetRolloutStuck fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
alert21.75%2 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 22: 3 · Sep 23: 1 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:37:02Z
  Ended:       2026-09-28T19:45:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:18:05Z
  Ended:       2026-09-28T13:31:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T13:18:05Z
  Ended:       2026-09-28T13:29:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71782104539007848288256
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert10.88%1 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 22: 2 · Sep 23: 2 · Sep 24: 4 · Sep 25: 6 · Sep 26: 1 · Sep 27: 2 · Sep 28: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:51:50Z
  Ended:       2026-09-28T06:57:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1154560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3rl3f8tj7aka16mnasqurrb5ebbvn0-w7g1w7z1p7g5a8x", pod="kube-apiserver-8448d69f6f-jqnnb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3rl3f8tj7aka16mnasqurrb5ebbvn0-w7g1w7z1p7g5a8x/kube-apiserver-8448d69f6f-jqnnb has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72022104441054601154560
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert10.88%1 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 22: 3 · Sep 23: 2 · Sep 24: 1 · Sep 25: 3 · Sep 26: 1 · Sep 27: 1 · Sep 28: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T21:44:52Z
  Ended:       2026-09-28T22:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T21:44:52Z
  Ended:       2026-09-28T22:00:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72292104669678952517632
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert10.88%1 of 114 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 22: 2 · Sep 23: 1 · Sep 24: 0 · Sep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
alert fired 16 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="cloud-network-config-controller-58dfb59986-bsjtm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2b4fc680-6a59-448e-b315-f78694553d9a"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller-58dfb59986-bsjtm (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="cloud-network-config-controller-69985fdf59-fmd5d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="387b4461-aa28-4131-85f9-f2b77c4e2e0e"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/cloud-network-config-controller-69985fdf59-fmd5d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="cloud-network-config-controller-69985fdf59-fmd5d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="387b4461-aa28-4131-85f9-f2b77c4e2e0e"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/cloud-network-config-controller-69985fdf59-fmd5d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="csi-snapshot-controller-d4d6cd4f4-d2tqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c3f26a0c-8e66-4f75-bb2e-f2645071d798"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/csi-snapshot-controller-d4d6cd4f4-d2tqr (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="csi-snapshot-controller-d4d6cd4f4-d2tqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c3f26a0c-8e66-4f75-bb2e-f2645071d798"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/csi-snapshot-controller-d4d6cd4f4-d2tqr (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="csi-snapshot-controller-c9cf48cfd-vfz96", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e625ddc5-e22f-4d5c-bb2f-d4cb602de924"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/csi-snapshot-controller-c9cf48cfd-vfz96 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="cloud-network-config-controller-5c87577d75-pzm4d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f7312b5e-d9af-4928-8d9a-ea34f1380f1e"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller-5c87577d75-pzm4d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="cloud-network-config-controller-5b9c45d5bc-245tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e7400d-4c21-4a6d-a049-fcf41ebbe075"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller-5b9c45d5bc-245tg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="cloud-network-config-controller-58dfb59986-bsjtm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2b4fc680-6a59-448e-b315-f78694553d9a"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller-58dfb59986-bsjtm (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="csi-snapshot-controller-69d89876cf-6p78f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a54f46c-0195-4d5e-aae8-68c97f711f84"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/csi-snapshot-controller-69d89876cf-6p78f (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="cloud-network-config-controller-5c87577d75-pzm4d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f7312b5e-d9af-4928-8d9a-ea34f1380f1e"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller-5c87577d75-pzm4d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="csi-snapshot-controller-69d89876cf-6p78f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a54f46c-0195-4d5e-aae8-68c97f711f84"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/csi-snapshot-controller-69d89876cf-6p78f (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="csi-snapshot-controller-7d6c8c6fbb-xmtc5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07ef49a6-b3c7-427b-ab6f-44204ea64710"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/csi-snapshot-controller-7d6c8c6fbb-xmtc5 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="csi-snapshot-controller-c9cf48cfd-vfz96", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e625ddc5-e22f-4d5c-bb2f-d4cb602de924"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/csi-snapshot-controller-c9cf48cfd-vfz96 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="cloud-network-config-controller-5b9c45d5bc-245tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e7400d-4c21-4a6d-a049-fcf41ebbe075"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller-5b9c45d5bc-245tg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="csi-snapshot-controller-7d6c8c6fbb-xmtc5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07ef49a6-b3c7-427b-ab6f-44204ea64710"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/csi-snapshot-controller-7d6c8c6fbb-xmtc5 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69002104485944341041152
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e1815.79%18 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 6 · Sep 26: 2 · Sep 27: 7 · Sep 28: 19INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs"
Unexpected error:
    <*fmt.wrapError | 0xc000f86d20>: 
    failed to create NodePool np-m8jhjs: failed waiting for nodepool="np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs" in resourcegroup="rg-candidate-4-23-f6rffb7vgk7k" to finish creating: GET https://rp.j5063808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e",
      "name": "451c3970-ee1e-443a-8418-d00eafee945e",
      "status": "Failed",
      "startTime": "2026-09-29T00:03:14.031479379Z",
      "endTime": "2026-09-29T00:22:18.567634171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs"
Unexpected error:
    <*fmt.wrapError | 0xc000f86d20>: 
    failed to create NodePool np-m8jhjs: failed waiting for nodepool="np-m8jhjs" for cluster "cluster-candidate-4-23-m8jhjs" in resourcegroup="rg-candidate-4-23-f6rffb7vgk7k" to finish creating: GET https://rp.j5063808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/451c3970-ee1e-443a-8418-d00eafee945e",
      "name": "451c3970-ee1e-443a-8418-d00eafee945e",
      "status": "Failed",
      "startTime": "2026-09-29T00:03:14.031479379Z",
      "endTime": "2026-09-29T00:22:18.567634171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-f6rffb7vgk7k-candidate-m8jhjs--managed/providers/Microsoft.Compute/virtualMachines/q4i8o9w4d0x3l0c-np-m8jhjs-4k5ks-9g9vh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000b234e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-nl2nr6" in resourcegroup="rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e",
      "name": "af93dcf2-82e0-4298-856d-f4846917b52e",
      "status": "Failed",
      "startTime": "2026-09-28T23:46:23.744577199Z",
      "endTime": "2026-09-29T00:05:30.871348708Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000b234e0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-nl2nr6" in resourcegroup="rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af93dcf2-82e0-4298-856d-f4846917b52e",
      "name": "af93dcf2-82e0-4298-856d-f4846917b52e",
      "status": "Failed",
      "startTime": "2026-09-28T23:46:23.744577199Z",
      "endTime": "2026-09-29T00:05:30.871348708Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nl2nr6-v9d65mgcz4rn-np-upgrade-nl-3d9177ea/providers/Microsoft.Compute/virtualMachines/m4b5p7t4y3f1r7l-npupgrade-4-20-glfwm-jbx5z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ff7440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-rdwv792csbm5" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "name": "bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "status": "Failed",
      "startTime": "2026-09-28T23:36:16.368959757Z",
      "endTime": "2026-09-28T23:55:24.990728156Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/virtualMachines/b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-d9rrr: Failed: availabilitysets failed to create or update. err: failed to get existing resource fips-enabled-rdwv792csbm5--managed/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/availabilitySets/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '1' seconds a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ff7440>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-rdwv792csbm5" to finish creating: GET https://rp.j8708096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "name": "bf2e602a-624e-40d9-b1d9-4619bec9a257",
      "status": "Failed",
      "startTime": "2026-09-28T23:36:16.368959757Z",
      "endTime": "2026-09-28T23:55:24.990728156Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/virtualMachines/b8s9z5t1h6u2q2n-np-1-b7tcg-zs7fm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8s9z5t1h6u2q2n-np-1-b7tcg-d9rrr: Failed: availabilitysets failed to create or update. err: failed to get existing resource fips-enabled-rdwv792csbm5--managed/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-rdwv792csbm5--managed/providers/Microsoft.Compute/availabilitySets/2t4ak80phdi6nre3sgcdj1b7rfktbt35_b8s9z5t1h6u2q2n-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '1' seconds a... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (14)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (18)
Date (UTC)Associated PRProw job
PR #72332104709786435063808
PR #69222104702432968708096
PR #71782104639037800189952
PR #72212104605288744095744
PR #71422104591950119178240
PR #71852104589634188087296
PR #72212104551522074365952
PR #69222104551505146155008
PR #71572104500979540955136
PR #71782104457246334259200
PR #71572104441018513362944
PR #68212104440974754189312
PR #62262104429033809776640
PR #71782104380048239759360
PR #72002104378704682553344
PR #70682104364976289353728
PR #72032104363449923407872
PR #70412104360746694479872
alert [hcp] userJourneyEtcdReadLatencyP991h5m fired
full failure pattern:
alert [hcp] userJourneyEtcdReadLatencyP991h5m fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
alert1210.53%12 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 6 · Sep 26: 5 · Sep 27: 8 · Sep 28: 12none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:05Z
  Ended:       2026-09-28T13:14:04Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:54:31Z
  Ended:       2026-09-28T12:18:28Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:24:54Z
  Ended:       2026-09-28T11:45:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP991h5m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdReadLatencyP991h5m does not fire12
Affected runs (12)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
PR #68842104529166530514944
PR #72182104519303637241856
PR #70682104500977863233536
PR #68842104493553664135168
PR #69002104485944341041152
PR #70712104484510623076352
PR #72142104466453632651264
PR #71782104457246334259200
PR #71782104421180684374016
PR #72122104397716221071360
PR #72032104363449923407872
alert [hcp] userJourneyEtcdReadLatencyP996h30m fired
full failure pattern:
alert [hcp] userJourneyEtcdReadLatencyP996h30m fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
alert1210.53%12 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 5 · Sep 26: 4 · Sep 27: 8 · Sep 28: 12none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:09:09Z
  Ended:       2026-09-28T13:23:06Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:57:33Z
  Ended:       2026-09-28T12:43:33Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:27:55Z
  Ended:       2026-09-28T12:10:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdReadLatencyP996h30m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-read-latency"
  Description: etcd gRPC Range (read) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdReadLatencyP996h30m does not fire12
Affected runs (12)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
PR #68842104529166530514944
PR #72182104519303637241856
PR #70682104500977863233536
PR #68842104493553664135168
PR #69002104485944341041152
PR #70712104484510623076352
PR #72142104466453632651264
PR #71782104457246334259200
PR #71782104421180684374016
PR #70682104364976289353728
PR #72032104363449923407872
ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, becaus...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code OverconstrainedZonalAllocationRequest; detail message Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\ - Availability Zone\\ - Differencing (Ephemeral) Disks\\ - Networking Constraints (such as Accelerated Networking or IPv6)\\ - VM Size\\; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision1210.53%12 of 114 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 12none
Full failure examples (3)
time=2026-09-28T22:45:44.302Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T22:53:20.758Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T22:45:44.312984171Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T22:45:46.293945727Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T22:45:46.372370612Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T22:53:20.755330702Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"93D44669EC2FD152\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft1\",\"resourceName\":\"ci01-j3781120-mgmt-1/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT2M24.7661361S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755440055Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"D5BB7C72C0AE0DC5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft2\",\"resourceName\":\"ci01-j3781120-mgmt-1/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT2M26.0225935S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755461175Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3C41555BEADB3DE5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/infra2\",\"resourceName\":\"ci01-j3781120-mgmt-1/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:45:49.7869316Z\",\"duration\":\"PT6M25.5435412S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755483916Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E73090DFF813B1FF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/infra1\",\"resourceName\":\"ci01-j3781120-mgmt-1/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT1M41.3605417S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755497506Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DB7E7572BE861E73\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft3\",\"resourceName\":\"ci01-j3781120-mgmt-1/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T22:50:52.9549448Z\",\"duration\":\"PT1M37.1165773S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"0\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T22:53:20.755594868Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A1922F5B4A5E438F\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1\",\"resourceName\":\"ci01-j3781120-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T22:45:49.7869316Z\",\"duration\":\"PT5M2.9744241S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T22:53:20.755630539Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3781120-mgmt-1\",\"clusterName\":\"ci01-j3781120-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109713389587325\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j3781120-mgmt-1/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 3, 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3781120-mgmt-1-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-25934385-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"0\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m36.442652668s\"}\n exit status 1"
time=2026-09-28T17:38:03.261Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T17:45:40.947Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T17:38:03.27081091Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T17:38:05.230722026Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T17:38:05.332101871Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T17:45:40.944777574Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"FDC79503A5389868\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j3078272-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT1M36.8422604S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"4\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 2, 1), 'Standard_E8ds_v6' (zones 1, 2), 'Standard_E8-4ds_v6' (zones 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T17:45:40.944878586Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DC248B23802810EC\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j3078272-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT1M22.2984424S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944892486Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F64F2B13FFA5C10A\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j3078272-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT2M10.1227436S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944904226Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"061A995FDE246942\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j3078272-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:38:09.8797046Z\",\"duration\":\"PT6M25.5728844S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944914146Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"4E79D1B051092E6E\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j3078272-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:43:12.5777514Z\",\"duration\":\"PT1M21.7193107S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944924847Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"E245F41D17E48B12\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2\",\"resourceName\":\"ci01-j3078272-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T17:38:09.8797046Z\",\"duration\":\"PT5M2.6044727S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T17:45:40.944945037Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j3078272-mgmt-2\",\"clusterName\":\"ci01-j3078272-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109897999722695\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j3078272-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss'. Please try alternative sizes 'Standard_D8s_v6' (zones 2, 1), 'Standard_E8ds_v6' (zones 1, 2), 'Standard_E8-4ds_v6' (zones 1, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3078272-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-41110440-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"4\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m37.674139897s\"}\n exit status 1"
time=2026-09-28T16:48:46.474Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-28T16:56:22.318Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-28T16:48:46.48313589Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-28T16:48:48.126456101Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-28T16:48:48.218251838Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-28T16:56:22.316196451Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"466F265DD247AC33\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j7409280-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:51.5604197Z\",\"duration\":\"PT7M11.5958475S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316251615Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"C2707792104BB37B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j7409280-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M38.4094883S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316265115Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"ED6706B9C7B9A085\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j7409280-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M54.5772258S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316275576Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9BFCDEDEF69AC484\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j7409280-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M20.8551291S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"OverconstrainedZonalAllocationRequest\",\"details\":[{\"code\":\"OverconstrainedZonalAllocationRequest\",\"message\":\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\",\"target\":\"3\"}],\"message\":\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-28T16:56:22.316323959Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"A08358469F613804\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j7409280-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:54:09.775917Z\",\"duration\":\"PT1M37.4251788S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.31633432Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F3D7EC585ED8A035\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2\",\"resourceName\":\"ci01-j7409280-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-28T16:48:51.5604197Z\",\"duration\":\"PT5M18.0784864S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-28T16:56:22.316354711Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7409280-mgmt-2\",\"clusterName\":\"ci01-j7409280-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584109927570685394\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"status\\\": \\\"Failed\\\",\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentFailed\\\",\\n    \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n    \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j7409280-mgmt-2/agentPools/userswft3\\\",\\n        \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n        \\\"details\\\": [\\n          {\\n            \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n            \\\"message\\\": \\\"Allocation failed on resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss'. Please try alternative sizes 'Standard_D16s_v6' (zones 2, 1), 'Standard_D16ds_v7' (zones 2), 'Standard_D16ads_v7' (zones 1, 3, 2) instead for higher likelihood of success. Read more about improving likelihood of operation success at https://aka.ms/AKSallocation. Create or update VMSS /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7409280-mgmt-2-aks1/providers/Microsoft.Compute/virtualMachineScaleSets/aks-userswft3-32404980-vmss failed. Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\",\\n            \\\"details\\\": [\\n              {\\n                \\\"code\\\": \\\"OverconstrainedZonalAllocationRequest\\\",\\n                \\\"target\\\": \\\"3\\\",\\n                \\\"message\\\": \\\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\\\n  - Availability Zone\\\\n  - Differencing (Ephemeral) Disks\\\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\\\n  - VM Size\\\\n\\\"\\n              }\\n            ]\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m35.833223151s\"}\n exit status 1"
Contributing tests (3)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)6
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)5
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster1
Affected runs (12)
Date (UTC)Associated PRProw job
PR #65772104696906683781120
PR #71782104618768113078272
PR #71692104608138077409280
PR #70742104605937833611264
PR #71692104592056927129600
PR #72152104592056415424512
PR #68692104582134038532096
PR #72002104579857332572160
PR #72152104576806710415360
PR #71692104540479675699200
PR #71692104526841447452672
PR #72172104491225984798720
alert [hcp] userJourneyEtcdWriteLatencyP991h5m fired
full failure pattern:
alert [hcp] userJourneyEtcdWriteLatencyP991h5m fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
alert108.77%10 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 5 · Sep 26: 3 · Sep 27: 5 · Sep 28: 10none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:06Z
  Ended:       2026-09-28T13:14:06Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:54:32Z
  Ended:       2026-09-28T12:18:31Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:41:58Z
  Ended:       2026-09-28T11:49:56Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP991h5m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="1h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="5m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 1h and 5m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdWriteLatencyP991h5m does not fire10
Affected runs (10)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
PR #68842104529166530514944
PR #72182104519303637241856
PR #71062104517513030144000
PR #69002104485944341041152
PR #72152104468013976653824
PR #71782104457246334259200
PR #70682104440930432978944
PR #71782104421180684374016
PR #72022104362505772994560
alert [hcp] userJourneyEtcdWriteLatencyP996h30m fired
full failure pattern:
alert [hcp] userJourneyEtcdWriteLatencyP996h30m fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
alert108.77%10 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 4 · Sep 26: 3 · Sep 27: 5 · Sep 28: 10none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:09:10Z
  Ended:       2026-09-28T13:39:13Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j8288256-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", region="centralus", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j8288256-mgmt-2
Namespace: ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:57:33Z
  Ended:       2026-09-28T12:43:33Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j0514944-mgmt-1", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a", region="canadacentral", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j0514944-mgmt-1
Namespace: ocm-arohcpci01-2t409ludea907cjj1c8fbcf54pvpn5p6-t9r3k6z8u9o6d0a
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T11:44:58Z
  Ended:       2026-09-28T12:14:55Z
  Severity:    Sev3
  Labels:      alertname="userJourneyEtcdWriteLatencyP996h30m", cluster="ci01-j7241856-mgmt-2", component="etcd", long_window="6h", namespace="ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas", region="westus3", severity="3", short_window="30m", slo="etcd-grpc-write-latency"
  Description: etcd gRPC Txn (write) P99 latency has exceeded 500ms in both the 6h and 30m windows.
Management Cluster: ci01-j7241856-mgmt-2
Namespace: ocm-arohcpci01-2t3vrficcj0hcpor2hrq2igqcsmsl70r-private-kas
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert userJourneyEtcdWriteLatencyP996h30m does not fire10
Affected runs (10)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
PR #68842104529166530514944
PR #72182104519303637241856
PR #71062104517513030144000
PR #69002104485944341041152
PR #72152104468013976653824
PR #71782104457246334259200
PR #70682104440930432978944
PR #71782104421180684374016
PR #72022104362505772994560
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e65.26%6 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 29none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002154570>: 
    resource group "complex-cilium-kv-jndv4cm2shzf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jndv4cm2shzf" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7",
      "name": "e40876ab-da07-40cc-bca8-2d36902295a7",
      "status": "Failed",
      "startTime": "2026-09-28T19:31:44.581294657Z",
      "endTime": "2026-09-28T19:55:45.796890355Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rtrm9n8ddf2ubgd68c1s64e1ioap still exists (deletion dispatched at 2026-09-28T19:31:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002154570>: 
    resource group "complex-cilium-kv-jndv4cm2shzf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jndv4cm2shzf" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e40876ab-da07-40cc-bca8-2d36902295a7",
      "name": "e40876ab-da07-40cc-bca8-2d36902295a7",
      "status": "Failed",
      "startTime": "2026-09-28T19:31:44.581294657Z",
      "endTime": "2026-09-28T19:55:45.796890355Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rtrm9n8ddf2ubgd68c1s64e1ioap still exists (deletion dispatched at 2026-09-28T19:31:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0013e4180>: 
    resource group "private-keyvault-bzhkhkm6vpw7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-bzhkhkm6vpw7" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1",
      "name": "486f914e-d8e9-4103-988d-8630bbe515a1",
      "status": "Failed",
      "startTime": "2026-09-28T19:29:30.639536976Z",
      "endTime": "2026-09-28T19:53:35.789612068Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rt8qnc76ptkosa2u3fv39hps3u1f still exists (deletion dispatched at 2026-09-28T19:29:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0013e4180>: 
    resource group "private-keyvault-bzhkhkm6vpw7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-bzhkhkm6vpw7" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/486f914e-d8e9-4103-988d-8630bbe515a1",
      "name": "486f914e-d8e9-4103-988d-8630bbe515a1",
      "status": "Failed",
      "startTime": "2026-09-28T19:29:30.639536976Z",
      "endTime": "2026-09-28T19:53:35.789612068Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46rt8qnc76ptkosa2u3fv39hps3u1f still exists (deletion dispatched at 2026-09-28T19:29:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f1a240>: 
    resource group "rg-candidate-4-23-8lfxrmnvvv2w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-8hk4r9" in resourcegroup="rg-candidate-4-23-8lfxrmnvvv2w" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4",
      "name": "7972cdda-000f-4549-95ba-7b26044f70b4",
      "status": "Failed",
      "startTime": "2026-09-28T19:49:57.362020562Z",
      "endTime": "2026-09-28T20:14:00.292519621Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46r6c80n9jpgk27jtqr1mg2ert1grv still exists (deletion dispatched at 2026-09-28T19:49:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000f1a240>: 
    resource group "rg-candidate-4-23-8lfxrmnvvv2w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-8hk4r9" in resourcegroup="rg-candidate-4-23-8lfxrmnvvv2w" to finish deleting: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7972cdda-000f-4549-95ba-7b26044f70b4",
      "name": "7972cdda-000f-4549-95ba-7b26044f70b4",
      "status": "Failed",
      "startTime": "2026-09-28T19:49:57.362020562Z",
      "endTime": "2026-09-28T20:14:00.292519621Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t46r6c80n9jpgk27jtqr1mg2ert1grv still exists (deletion dispatched at 2026-09-28T19:49:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (23)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.234
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (6)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71782104539007848288256
PR #69002104485944341041152
PR #71782104457246334259200
PR #71782104421180684374016
PR #71782104380048239759360
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e32.63%3 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 3 · Sep 26: 0 · Sep 27: 0 · Sep 28: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c241a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-c8jw5nmfkndx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808",
      "name": "4538241d-2021-40b7-8228-fb4052b1f808",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:37.948094059Z",
      "endTime": "2026-09-28T19:39:45.568048903Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000c241a0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-c8jw5nmfkndx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4538241d-2021-40b7-8228-fb4052b1f808",
      "name": "4538241d-2021-40b7-8228-fb4052b1f808",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:37.948094059Z",
      "endTime": "2026-09-28T19:39:45.568048903Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--gd5x92vx9cnh/cluster-hs-pre-r2s4dx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fd2480>: 
    failed to create HCP cluster cluster-hs-pre-r2s4dx: failed waiting for cluster="cluster-hs-pre-r2s4dx" in resourcegroup="rg-hs-pre--gd5x92vx9cnh" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "name": "471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:54.032352928Z",
      "endTime": "2026-09-28T13:23:00.490629254Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--gd5x92vx9cnh/cluster-hs-pre-r2s4dx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fd2480>: 
    failed to create HCP cluster cluster-hs-pre-r2s4dx: failed waiting for cluster="cluster-hs-pre-r2s4dx" in resourcegroup="rg-hs-pre--gd5x92vx9cnh" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "name": "471a7bfe-fca1-453d-8b53-ca50a511f4b4",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:54.032352928Z",
      "endTime": "2026-09-28T13:23:00.490629254Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 22m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000dbafa0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-xh7nfd4qdxnv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a",
      "name": "10cf3d15-2823-4e15-9280-d4bdf793742a",
      "status": "Failed",
      "startTime": "2026-09-28T12:54:54.12824002Z",
      "endTime": "2026-09-28T13:24:00.287293188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000dbafa0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-xh7nfd4qdxnv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/10cf3d15-2823-4e15-9280-d4bdf793742a",
      "name": "10cf3d15-2823-4e15-9280-d4bdf793742a",
      "status": "Failed",
      "startTime": "2026-09-28T12:54:54.12824002Z",
      "endTime": "2026-09-28T13:24:00.287293188Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m25s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71782104539007848288256
PR #71782104457246334259200
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e32.63%3 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc001512f00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-5f68mkqs6vtp" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-5f68mkqs6vtp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc001512f00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-5f68mkqs6vtp" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-5f68mkqs6vtp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc0033b6750>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-nh4vk2wgfp5z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-nh4vk2wgfp5z, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc0033b6750>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-nh4vk2wgfp5z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-nh4vk2wgfp5z, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc003167da0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kg9lzmtgpr42" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kg9lzmtgpr42, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc003167da0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kg9lzmtgpr42" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kg9lzmtgpr42, error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling2
Affected runs (3)
Date (UTC)Associated PRProw job
PR #71412104641856926126080
PR #72122104397716221071360
PR #71572104364925383086080
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e21.75%2 of 114 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-x7z7b7"
Unexpected error:
    <*fmt.wrapError | 0xc000bab020>: 
    DNS for route host agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-x7z7b7"
Unexpected error:
    <*fmt.wrapError | 0xc000bab020>: 
    DNS for route host agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000c3c580>: 
    DNS for route host agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000c3c580>: 
    DNS for route host agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #68692104433568867422208
PR #62262104429033809776640

Environment: INT

Failure patterns: 1 · Failures matched: 1 · Total runs: 3 · Runs affected: 1

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e133.33%1 of 3 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 1DEV, PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb"
Unexpected error:
    <*fmt.wrapError | 0xc00024f2c0>: 
    failed to create NodePool np-fnv7lb: failed waiting for nodepool="np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb" in resourcegroup="rg-candidate-5-0-v6nvmt4rxzjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a",
      "name": "203ad465-1e1f-4337-9659-f5fe2972a72a",
      "status": "Failed",
      "startTime": "2026-09-28T21:17:06.138034027Z",
      "endTime": "2026-09-28T21:36:06.809868093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb"
Unexpected error:
    <*fmt.wrapError | 0xc00024f2c0>: 
    failed to create NodePool np-fnv7lb: failed waiting for nodepool="np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb" in resourcegroup="rg-candidate-5-0-v6nvmt4rxzjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a",
      "name": "203ad465-1e1f-4337-9659-f5fe2972a72a",
      "status": "Failed",
      "startTime": "2026-09-28T21:17:06.138034027Z",
      "endTime": "2026-09-28T21:36:06.809868093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104667201544916992

Environment: PROD

Failure patterns: 9 · Failures matched: 26 · Total runs: 13 · Runs affected: 7

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e323.08%3 of 13 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007275a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdcdkl9rvn5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612",
      "name": "3a70e081-8150-479f-812b-31eb02522612",
      "status": "Failed",
      "startTime": "2026-09-29T01:01:02.974562787Z",
      "endTime": "2026-09-29T01:20:06.931508754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007275a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdcdkl9rvn5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612",
      "name": "3a70e081-8150-479f-812b-31eb02522612",
      "status": "Failed",
      "startTime": "2026-09-29T01:01:02.974562787Z",
      "endTime": "2026-09-29T01:20:06.931508754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc001055c80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tf5tm29lbrgl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "name": "51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "status": "Failed",
      "startTime": "2026-09-28T23:10:33.615095623Z",
      "endTime": "2026-09-28T23:29:40.114080242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2fhfr-24lbg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2fhfr-24lbg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc001055c80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tf5tm29lbrgl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "name": "51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "status": "Failed",
      "startTime": "2026-09-28T23:10:33.615095623Z",
      "endTime": "2026-09-28T23:29:40.114080242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2fhfr-24lbg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2fhfr-24lbg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0030a64a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kx45mg6gnp52" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b",
      "name": "7aa70862-e406-44df-915d-27100318e68b",
      "status": "Failed",
      "startTime": "2026-09-28T23:56:08.330966898Z",
      "endTime": "2026-09-29T00:15:12.920582798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0030a64a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kx45mg6gnp52" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b",
      "name": "7aa70862-e406-44df-915d-27100318e68b",
      "status": "Failed",
      "startTime": "2026-09-28T23:56:08.330966898Z",
      "endTime": "2026-09-29T00:15:12.920582798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2104710565229236224
n/a2104707837916614656
n/a2104576811751968768
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
e2e17.69%1 of 13 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 22: 5 · Sep 23: 4 · Sep 24: 5 · Sep 25: 5 · Sep 26: 5 · Sep 27: 0 · Sep 28: 5STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a51a10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mnwb4zfspsdb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a51a10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mnwb4zfspsdb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00179cc60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-v4rx8l2tw9wq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00179cc60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-v4rx8l2tw9wq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000aa43c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qgqdfprzx89s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000aa43c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qgqdfprzx89s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837916614656
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e17.69%1 of 13 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 22: 4 · Sep 23: 4 · Sep 24: 4 · Sep 25: 4 · Sep 26: 2 · Sep 27: 4 · Sep 28: 4STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001048440>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-k6t82b" resourcegroup="rg-np-dg-minor-k6t82b-4f2hjngd28k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-k6t82b-4f2hjngd28k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-k6t82b/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001048440>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-k6t82b" resourcegroup="rg-np-dg-minor-k6t82b-4f2hjngd28k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-k6t82b-4f2hjngd28k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-k6t82b/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000ade640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-t7jl46" in resourcegroup="rg-np-version-upgrade-t7jl46-65khsnvbc4zl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t7jl46-65khsnvbc4zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t7jl46/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000ade640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-t7jl46" in resourcegroup="rg-np-version-upgrade-t7jl46-65khsnvbc4zl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t7jl46-65khsnvbc4zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t7jl46/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9"
Unexpected error:
    <*fmt.wrapError | 0xc000d24340>: 
    failed to create NodePool np-5cw4w9: failed starting nodepool creation "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9" in resourcegroup="rg-nightly-4-20-b7gtl2svm55n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-b7gtl2svm55n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-5cw4w9/nodePools/np-5cw4w9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9"
Unexpected error:
    <*fmt.wrapError | 0xc000d24340>: 
    failed to create NodePool np-5cw4w9: failed starting nodepool creation "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9" in resourcegroup="rg-nightly-4-20-b7gtl2svm55n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-b7gtl2svm55n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-5cw4w9/nodePools/np-5cw4w9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837916614656
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e17.69%1 of 13 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 22: 1 · Sep 23: 1 · Sep 24: 1 · Sep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f35488>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f35488>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837916614656
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e430.77%4 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4DEV, INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m"
Unexpected error:
    <*fmt.wrapError | 0xc00087e120>: 
    failed to create NodePool np-rcls6m: failed waiting for nodepool="np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m" in resourcegroup="rg-candidate-4-21-92w77dwp5rg8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "name": "a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:31.628142216Z",
      "endTime": "2026-09-29T00:35:38.838060326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m"
Unexpected error:
    <*fmt.wrapError | 0xc00087e120>: 
    failed to create NodePool np-rcls6m: failed waiting for nodepool="np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m" in resourcegroup="rg-candidate-4-21-92w77dwp5rg8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "name": "a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:31.628142216Z",
      "endTime": "2026-09-29T00:35:38.838060326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc00130ef60>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-7g7qln9l2np5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "name": "ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "status": "Failed",
      "startTime": "2026-09-28T23:42:15.584467675Z",
      "endTime": "2026-09-29T00:01:22.906593334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc00130ef60>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-7g7qln9l2np5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "name": "ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "status": "Failed",
      "startTime": "2026-09-28T23:42:15.584467675Z",
      "endTime": "2026-09-29T00:01:22.906593334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l"
Unexpected error:
    <*fmt.wrapError | 0xc00171b4a0>: 
    failed to create NodePool np-nmsq2l: failed waiting for nodepool="np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l" in resourcegroup="rg-candidate-4-21-lf4lgs4pzp7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975",
      "name": "8303805f-b000-457e-b724-d4df7eac6975",
      "status": "Failed",
      "startTime": "2026-09-28T21:30:59.759776246Z",
      "endTime": "2026-09-28T21:50:00.145843425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l"
Unexpected error:
    <*fmt.wrapError | 0xc00171b4a0>: 
    failed to create NodePool np-nmsq2l: failed waiting for nodepool="np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l" in resourcegroup="rg-candidate-4-21-lf4lgs4pzp7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975",
      "name": "8303805f-b000-457e-b724-d4df7eac6975",
      "status": "Failed",
      "startTime": "2026-09-28T21:30:59.759776246Z",
      "endTime": "2026-09-28T21:50:00.145843425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2104713411836252160
n/a2104707837916614656
n/a2104669292057333760
n/a2104667835979534336
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or q...
full failure pattern:
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e215.38%2 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc000b43f40>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc000b43f40>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001139d00>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001139d00>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2104713411836252160
n/a2104667835979534336
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b"
Unexpected error:
    <*fmt.wrapError | 0xc000b7f080>: 
    failed to create NodePool np-bhxv8b: failed waiting for nodepool="np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b" in resourcegroup="rg-candidate-4-21-mrwxnrjtcxks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "name": "b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:53.990233971Z",
      "endTime": "2026-09-29T00:35:58.987434024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b"
Unexpected error:
    <*fmt.wrapError | 0xc000b7f080>: 
    failed to create NodePool np-bhxv8b: failed waiting for nodepool="np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b" in resourcegroup="rg-candidate-4-21-mrwxnrjtcxks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "name": "b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:53.990233971Z",
      "endTime": "2026-09-29T00:35:58.987434024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000bad000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-64htmh" in resourcegroup="rg-np-noedge-64htmh-c4j9qdqfrq4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "name": "3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "status": "Failed",
      "startTime": "2026-09-29T01:00:25.564607872Z",
      "endTime": "2026-09-29T01:19:27.08336887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: Creating: virtualmachine creating or updating\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000bad000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-64htmh" in resourcegroup="rg-np-noedge-64htmh-c4j9qdqfrq4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "name": "3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "status": "Failed",
      "startTime": "2026-09-29T01:00:25.564607872Z",
      "endTime": "2026-09-29T01:19:27.08336887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: Creating: virtualmachine creating or updating\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb"
Unexpected error:
    <*fmt.wrapError | 0xc0010410e0>: 
    failed to create NodePool np-5m4wlb: failed waiting for nodepool="np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb" in resourcegroup="rg-candidate-4-23-wwwsqs8cghpq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052",
      "name": "91d9873c-e44e-4e44-ba41-ade87f500052",
      "status": "Failed",
      "startTime": "2026-09-29T00:18:19.018330415Z",
      "endTime": "2026-09-29T00:37:28.829514453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: Creating: virtualmachine creating or updating\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb"
Unexpected error:
    <*fmt.wrapError | 0xc0010410e0>: 
    failed to create NodePool np-5m4wlb: failed waiting for nodepool="np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb" in resourcegroup="rg-candidate-4-23-wwwsqs8cghpq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052",
      "name": "91d9873c-e44e-4e44-ba41-ade87f500052",
      "status": "Failed",
      "startTime": "2026-09-29T00:18:19.018330415Z",
      "endTime": "2026-09-29T00:37:28.829514453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: Creating: virtualmachine creating or updating\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104710565229236224
storage account log verification failed
full failure pattern:
storage account log verification failed
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 4 · Sep 25: 2 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4z25qr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7lft27tlvnbf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0009fab20>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4z25qr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7lft27tlvnbf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0009fab20>: 
    storage account log verification failed
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104390744759341056
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet po...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-q9zwpnrqm4zb/cluster-nightly-5-1-sq64f4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110cda0>: 
    failed to create HCP cluster cluster-nightly-5-1-sq64f4: failed waiting for cluster="cluster-nightly-5-1-sq64f4" in resourcegroup="rg-nightly-5-1-q9zwpnrqm4zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec",
      "name": "25287b3f-b870-4e18-a51f-514432892aec",
      "status": "Failed",
      "startTime": "2026-09-28T23:31:45.913312Z",
      "endTime": "2026-09-29T00:00:52.759469958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-q9zwpnrqm4zb/cluster-nightly-5-1-sq64f4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110cda0>: 
    failed to create HCP cluster cluster-nightly-5-1-sq64f4: failed waiting for cluster="cluster-nightly-5-1-sq64f4" in resourcegroup="rg-nightly-5-1-q9zwpnrqm4zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec",
      "name": "25287b3f-b870-4e18-a51f-514432892aec",
      "status": "Failed",
      "startTime": "2026-09-28T23:31:45.913312Z",
      "endTime": "2026-09-29T00:00:52.759469958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837916614656

Environment: STG

Failure patterns: 6 · Failures matched: 14 · Total runs: 4 · Runs affected: 2

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e125.00%1 of 4 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 5 · Sep 25: 5 · Sep 26: 5 · Sep 27: 0 · Sep 28: 5PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a097a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-swk5ggttxwjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a097a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-swk5ggttxwjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00160fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-hfkhgg4cj7jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00160fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-hfkhgg4cj7jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000b459e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-sf9f9x57dtfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000b459e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-sf9f9x57dtfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837958557696
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e125.00%1 of 4 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 3 · Sep 25: 4 · Sep 26: 2 · Sep 27: 1 · Sep 28: 4PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001432880>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lllf2h" resourcegroup="rg-np-dg-minor-lllf2h-lmftzgmmkgfx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lllf2h-lmftzgmmkgfx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lllf2h/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001432880>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lllf2h" resourcegroup="rg-np-dg-minor-lllf2h-lmftzgmmkgfx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lllf2h-lmftzgmmkgfx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lllf2h/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x"
Unexpected error:
    <*fmt.wrapError | 0xc0008e6340>: 
    failed to create NodePool np-br2c4x: failed starting nodepool creation "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x" in resourcegroup="rg-nightly-4-20-bvth8x7dzstx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-bvth8x7dzstx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-br2c4x/nodePools/np-br2c4x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x"
Unexpected error:
    <*fmt.wrapError | 0xc0008e6340>: 
    failed to create NodePool np-br2c4x: failed starting nodepool creation "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x" in resourcegroup="rg-nightly-4-20-bvth8x7dzstx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-bvth8x7dzstx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-br2c4x/nodePools/np-br2c4x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000dbaaa0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-grgfk2" in resourcegroup="rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-grgfk2/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000dbaaa0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-grgfk2" in resourcegroup="rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-grgfk2/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837958557696
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e125.00%1 of 4 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d235a8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d235a8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837958557696
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 17 · Sep 27: 1 · Sep 28: 2DEV, INT, PROD
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8"
Unexpected error:
    <*fmt.wrapError | 0xc0012916a0>: 
    failed to create NodePool np-dpvlx8: failed waiting for nodepool="np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8" in resourcegroup="rg-nightly-5-0-5jrsqj92ljxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e",
      "name": "eee4d653-3b34-478d-9851-5151f132ff5e",
      "status": "Failed",
      "startTime": "2026-09-29T00:23:56.849718476Z",
      "endTime": "2026-09-29T00:43:01.619582837Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8"
Unexpected error:
    <*fmt.wrapError | 0xc0012916a0>: 
    failed to create NodePool np-dpvlx8: failed waiting for nodepool="np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8" in resourcegroup="rg-nightly-5-0-5jrsqj92ljxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e",
      "name": "eee4d653-3b34-478d-9851-5151f132ff5e",
      "status": "Failed",
      "startTime": "2026-09-29T00:23:56.849718476Z",
      "endTime": "2026-09-29T00:43:01.619582837Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001873458>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zwldz5957z9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "name": "00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "status": "Failed",
      "startTime": "2026-09-28T23:57:06.99965356Z",
      "endTime": "2026-09-29T00:16:13.233773533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-lwwnr-xksft: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-lwwnr-xksft: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001873458>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zwldz5957z9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "name": "00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "status": "Failed",
      "startTime": "2026-09-28T23:57:06.99965356Z",
      "endTime": "2026-09-29T00:16:13.233773533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-lwwnr-xksft: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-lwwnr-xksft: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837958557696
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet po...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 6 · Sep 27: 41 · Sep 28: 1PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdf28j9mzrjf/cluster-nightly-5-1-ds89w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a3e960>: 
    failed to create HCP cluster cluster-nightly-5-1-ds89w5: failed waiting for cluster="cluster-nightly-5-1-ds89w5" in resourcegroup="rg-nightly-5-1-vdf28j9mzrjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b",
      "name": "4cc98024-5165-40da-b846-ec5ee553964b",
      "status": "Failed",
      "startTime": "2026-09-29T00:13:16.920404751Z",
      "endTime": "2026-09-29T00:42:21.556617536Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdf28j9mzrjf/cluster-nightly-5-1-ds89w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a3e960>: 
    failed to create HCP cluster cluster-nightly-5-1-ds89w5: failed waiting for cluster="cluster-nightly-5-1-ds89w5" in resourcegroup="rg-nightly-5-1-vdf28j9mzrjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b",
      "name": "4cc98024-5165-40da-b846-ec5ee553964b",
      "status": "Failed",
      "startTime": "2026-09-29T00:13:16.920404751Z",
      "endTime": "2026-09-29T00:42:21.556617536Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837958557696
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001152660>: 
    resource group "rg-cp-ystream-upgrade-4-22-k576cn8fjplq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-c95x5l" in resourcegroup="rg-cp-ystream-upgrade-4-22-k576cn8fjplq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "name": "5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "status": "Failed",
      "startTime": "2026-09-28T22:54:48.090437049Z",
      "endTime": "2026-09-28T23:18:56.801422937Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t49njjvu97eroh1p7j9jssq535c6590 still exists (deletion dispatched at 2026-09-28T22:54:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001152660>: 
    resource group "rg-cp-ystream-upgrade-4-22-k576cn8fjplq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-c95x5l" in resourcegroup="rg-cp-ystream-upgrade-4-22-k576cn8fjplq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "name": "5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "status": "Failed",
      "startTime": "2026-09-28T22:54:48.090437049Z",
      "endTime": "2026-09-28T23:18:56.801422937Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t49njjvu97eroh1p7j9jssq535c6590 still exists (deletion dispatched at 2026-09-28T22:54:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104687818398765056