CIHealth
<
Last 7 Days▾
Last 7 Days Weekly: Sep 28 - Oct 4 Sprint: Sep 28 - Oct 11 Single Day: Sep 28
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 164 · Failures matched: 1349 · Total runs: 707 · Runs affected: 549

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e20.28%2 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000911660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6qt82pv5cb4k" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088",
      "name": "cd024c78-818b-4b49-8634-2f704d9a8088",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:28.191649845Z",
      "endTime": "2026-09-28T07:47:33.272794005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000911660>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-6qt82pv5cb4k" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd024c78-818b-4b49-8634-2f704d9a8088",
      "name": "cd024c78-818b-4b49-8634-2f704d9a8088",
      "status": "Failed",
      "startTime": "2026-09-28T07:18:28.191649845Z",
      "endTime": "2026-09-28T07:47:33.272794005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001698220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hqr7q7xq4llr" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "name": "e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:08.942854859Z",
      "endTime": "2026-09-28T05:23:14.16072165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001698220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hqr7q7xq4llr" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "name": "e46d4d39-f739-4c7b-9f6e-359435d906f6",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:08.942854859Z",
      "endTime": "2026-09-28T05:23:14.16072165Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0005d2ce0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sbtnmq6dk9ch" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758",
      "name": "1f13de10-f443-42eb-b391-6417a9194758",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:46.663436186Z",
      "endTime": "2026-09-28T05:24:54.296612042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0005d2ce0>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-sbtnmq6dk9ch" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1f13de10-f443-42eb-b391-6417a9194758",
      "name": "1f13de10-f443-42eb-b391-6417a9194758",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:46.663436186Z",
      "endTime": "2026-09-28T05:24:54.296612042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: router deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782104457246334259200
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a6bae0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5w2hj7t2fgmv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "name": "fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:32.768656716Z",
      "endTime": "2026-09-28T13:22:40.579367905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m26s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000a6bae0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-5w2hj7t2fgmv" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "name": "fb21162c-bce0-4144-b0c6-c225f21cfacb",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:32.768656716Z",
      "endTime": "2026-09-28T13:22:40.579367905Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m26s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0xc000ea49c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-q4hdh7csw5qs" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508",
      "name": "daab790a-4984-45ab-a2a3-a3f37cd79508",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:48.885600114Z",
      "endTime": "2026-09-28T13:22:50.894287651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m10s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0xc000ea49c0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-q4hdh7csw5qs" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/daab790a-4984-45ab-a2a3-a3f37cd79508",
      "name": "daab790a-4984-45ab-a2a3-a3f37cd79508",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:48.885600114Z",
      "endTime": "2026-09-28T13:22:50.894287651Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m10s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d60180>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-txbk9bmdh8wg" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "name": "7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:22.508354202Z",
      "endTime": "2026-09-28T13:22:30.229773982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 26m20s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d60180>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-txbk9bmdh8wg" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "name": "7ab8bf70-a76d-451a-a68e-55fd5d0807a5",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:22.508354202Z",
      "endTime": "2026-09-28T13:22:30.229773982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 26m20s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2s722b
Unexpected error:
    <*fmt.wrapError | 0xc000bfeb60>: 
    failed to create HCP cluster np-noedge-2s722b: failed waiting for cluster="np-noedge-2s722b" in resourcegroup="rg-np-noedge-2s722b-ttwnzxstsj4p" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93",
      "name": "18b3752b-dfe7-413c-83e0-e4570361ec93",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:09.735885483Z",
      "endTime": "2026-09-28T05:23:14.561838226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2s722b
Unexpected error:
    <*fmt.wrapError | 0xc000bfeb60>: 
    failed to create HCP cluster np-noedge-2s722b: failed waiting for cluster="np-noedge-2s722b" in resourcegroup="rg-np-noedge-2s722b-ttwnzxstsj4p" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/18b3752b-dfe7-413c-83e0-e4570361ec93",
      "name": "18b3752b-dfe7-413c-83e0-e4570361ec93",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:09.735885483Z",
      "endTime": "2026-09-28T05:23:14.561838226Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000fc65a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-6vnwh8vjnz8r" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f",
      "name": "0b3163de-a897-4f45-9b77-e785bdfab09f",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.798033495Z",
      "endTime": "2026-09-28T05:22:54.610455796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc000fc65a0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-6vnwh8vjnz8r" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0b3163de-a897-4f45-9b77-e785bdfab09f",
      "name": "0b3163de-a897-4f45-9b77-e785bdfab09f",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.798033495Z",
      "endTime": "2026-09-28T05:22:54.610455796Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8fvg54" with version 4.22.15 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ae0680>: 
    failed to create HCP cluster cluster-zstream-4-22-8fvg54: failed waiting for cluster="cluster-zstream-4-22-8fvg54" in resourcegroup="rg-zstream-upgrade-4-22-8zfwm76xgnlj" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0",
      "name": "847235cf-8889-474c-8fc4-418de69191f0",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:18.057920805Z",
      "endTime": "2026-09-28T19:39:25.918712924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8fvg54" with version 4.22.15 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ae0680>: 
    failed to create HCP cluster cluster-zstream-4-22-8fvg54: failed waiting for cluster="cluster-zstream-4-22-8fvg54" in resourcegroup="rg-zstream-upgrade-4-22-8zfwm76xgnlj" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/847235cf-8889-474c-8fc4-418de69191f0",
      "name": "847235cf-8889-474c-8fc4-418de69191f0",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:18.057920805Z",
      "endTime": "2026-09-28T19:39:25.918712924Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-27p526" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d20a60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-27p526: failed waiting for cluster="cp-ystream-upgrade-4-22-27p526" in resourcegroup="rg-cp-ystream-upgrade-4-22-7drm72sqb8zr" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "name": "3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:49.293974095Z",
      "endTime": "2026-09-28T19:39:55.897408712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-27p526" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000d20a60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-27p526: failed waiting for cluster="cp-ystream-upgrade-4-22-27p526" in resourcegroup="rg-cp-ystream-upgrade-4-22-7drm72sqb8zr" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "name": "3b089068-0a7a-4b3d-bdf3-9f03ebedd6b2",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:49.293974095Z",
      "endTime": "2026-09-28T19:39:55.897408712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000ce94a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j9cxlp" in resourcegroup="rg-np-noedge-j9cxlp-9zvfnm962trj" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "name": "3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "status": "Failed",
      "startTime": "2026-09-28T17:41:00.477362299Z",
      "endTime": "2026-09-28T18:00:09.600349173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000ce94a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j9cxlp" in resourcegroup="rg-np-noedge-j9cxlp-9zvfnm962trj" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "name": "3c1e081e-0b8b-4062-9367-d3e3f4309da6",
      "status": "Failed",
      "startTime": "2026-09-28T17:41:00.477362299Z",
      "endTime": "2026-09-28T18:00:09.600349173Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z"
Unexpected error:
    <*fmt.wrapError | 0xc000f712c0>: 
    failed to create NodePool np-wsvb2z: failed waiting for nodepool="np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z" in resourcegroup="rg-candidate-4-20-slrjsf69bdm7" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7",
      "name": "119c8117-1a73-44d6-8ce8-38b6990193b7",
      "status": "Failed",
      "startTime": "2026-09-28T17:43:51.079865069Z",
      "endTime": "2026-09-28T18:02:59.523800413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z"
Unexpected error:
    <*fmt.wrapError | 0xc000f712c0>: 
    failed to create NodePool np-wsvb2z: failed waiting for nodepool="np-wsvb2z" for cluster "cluster-candidate-4-20-wsvb2z" in resourcegroup="rg-candidate-4-20-slrjsf69bdm7" to finish creating: GET https://rp.j7194880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/119c8117-1a73-44d6-8ce8-38b6990193b7",
      "name": "119c8117-1a73-44d6-8ce8-38b6990193b7",
      "status": "Failed",
      "startTime": "2026-09-28T17:43:51.079865069Z",
      "endTime": "2026-09-28T18:02:59.523800413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71242104613528387194880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal: Timed out after <duration>s.
full failure pattern:
Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal: Timed out after <duration>s.
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:164 with:
expected HTTP 200 from sample app via internal LB, got 503
Expected
    <string>: ...
to equal
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:164 with:
expected HTTP 200 from sample app via internal LB, got 503
Expected
    <string>: ...
to equal
    <string>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-apiserver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-26r6nt" with version 4.20.39 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106e580>: 
    failed to create HCP cluster cluster-zstream-4-20-26r6nt: failed waiting for cluster="cluster-zstream-4-20-26r6nt" in resourcegroup="rg-zstream-upgrade-4-20-6bpd6fpsn6s4" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "name": "4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:43.832400833Z",
      "endTime": "2026-09-28T13:22:51.48967289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-apiserver, openshift-route-controller-manager, catalog-operator, community-operators-catalog, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-storage-operator, openshift-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, ingress-operator, cluster-policy-controller, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-26r6nt" with version 4.20.39 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106e580>: 
    failed to create HCP cluster cluster-zstream-4-20-26r6nt: failed waiting for cluster="cluster-zstream-4-20-26r6nt" in resourcegroup="rg-zstream-upgrade-4-20-6bpd6fpsn6s4" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "name": "4dd629ed-2a31-430a-9bfc-f3b253048b0d",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:43.832400833Z",
      "endTime": "2026-09-28T13:22:51.48967289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-apiserver, openshift-route-controller-manager, catalog-operator, community-operators-catalog, hosted-cluster-config-operator, olm-operator, cluster-version-operator, cluster-storage-operator, openshift-controller-manager, dns-operator, packageserver, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, ingress-operator, cluster-policy-controller, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc0010c41e0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-jfwb2dgmj26t" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "name": "a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:17.858913168Z",
      "endTime": "2026-09-28T05:23:24.118546984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc0010c41e0>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-jfwb2dgmj26t" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "name": "a6a2f8c2-ae4c-4f51-8a9f-44a84ab3a613",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:17.858913168Z",
      "endTime": "2026-09-28T05:23:24.118546984Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, kube-apiserver, openshift-apiserver, packageserver, redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010432a0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-66ncz7vqwm2z" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86",
      "name": "c35d648d-395d-429f-9f2d-24eed4314d86",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:11.81451702Z",
      "endTime": "2026-09-28T19:39:16.076223121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, community-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:99]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010432a0>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-66ncz7vqwm2z" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35d648d-395d-429f-9f2d-24eed4314d86",
      "name": "c35d648d-395d-429f-9f2d-24eed4314d86",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:11.81451702Z",
      "endTime": "2026-09-28T19:39:16.076223121Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, community-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6kj9nw5mjrc2/cluster-hs-pre-fvbpxs should provision
Unexpected error:
    <*fmt.wrapError | 0xc000db85e0>: 
    failed to create HCP cluster cluster-hs-pre-fvbpxs: failed waiting for cluster="cluster-hs-pre-fvbpxs" in resourcegroup="rg-hs-pre--6kj9nw5mjrc2" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd",
      "name": "2f78ab67-468f-449c-890b-25423489ffcd",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.674200437Z",
      "endTime": "2026-09-28T05:23:04.406727689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6kj9nw5mjrc2/cluster-hs-pre-fvbpxs should provision
Unexpected error:
    <*fmt.wrapError | 0xc000db85e0>: 
    failed to create HCP cluster cluster-hs-pre-fvbpxs: failed waiting for cluster="cluster-hs-pre-fvbpxs" in resourcegroup="rg-hs-pre--6kj9nw5mjrc2" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2f78ab67-468f-449c-890b-25423489ffcd",
      "name": "2f78ab67-468f-449c-890b-25423489ffcd",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.674200437Z",
      "endTime": "2026-09-28T05:23:04.406727689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, certified-operators-catalog, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ae31a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-5cbtfzkp497x" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "name": "6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:33.784163345Z",
      "endTime": "2026-09-28T05:22:34.354425424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ae31a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-5cbtfzkp497x" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "name": "6576461c-4ae7-4100-90f9-94b3c7ae31de",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:33.784163345Z",
      "endTime": "2026-09-28T05:22:34.354425424Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cb0840>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rbwvmrnzw8j5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e",
      "name": "5bdae3db-e612-4f27-8693-02200538e08e",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:23.74994885Z",
      "endTime": "2026-09-28T05:23:24.026384001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000cb0840>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rbwvmrnzw8j5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5bdae3db-e612-4f27-8693-02200538e08e",
      "name": "5bdae3db-e612-4f27-8693-02200538e08e",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:23.74994885Z",
      "endTime": "2026-09-28T05:23:24.026384001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [community-operators-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00106c320>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-qdqkz9rfmldx" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23",
      "name": "0c723202-68f5-41c7-a5d4-762157d31d23",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.997312592Z",
      "endTime": "2026-09-28T05:22:54.143428969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc00106c320>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-qdqkz9rfmldx" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0c723202-68f5-41c7-a5d4-762157d31d23",
      "name": "0c723202-68f5-41c7-a5d4-762157d31d23",
      "status": "Failed",
      "startTime": "2026-09-28T04:53:53.997312592Z",
      "endTime": "2026-09-28T05:22:54.143428969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: router; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gng5crkc9wh8/cluster-candidate-4-22-w8wr72 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00117a200>: 
    failed to create HCP cluster cluster-candidate-4-22-w8wr72: failed waiting for cluster="cluster-candidate-4-22-w8wr72" in resourcegroup="rg-candidate-4-22-gng5crkc9wh8" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "name": "00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.970731237Z",
      "endTime": "2026-09-28T05:23:04.188577598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, router; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gng5crkc9wh8/cluster-candidate-4-22-w8wr72 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00117a200>: 
    failed to create HCP cluster cluster-candidate-4-22-w8wr72: failed waiting for cluster="cluster-candidate-4-22-w8wr72" in resourcegroup="rg-candidate-4-22-gng5crkc9wh8" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "name": "00770e12-5d2a-48dc-92a8-bd5616100b1d",
      "status": "Failed",
      "startTime": "2026-09-28T04:54:03.970731237Z",
      "endTime": "2026-09-28T05:23:04.188577598Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, router; hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint <hcp-api-host> failed: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: [redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-rxbdcb" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001090b40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-rxbdcb: failed waiting for cluster="cp-ystream-upgrade-4-23-rxbdcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-jvcp8dmnbrt7" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "name": "1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:56.829077997Z",
      "endTime": "2026-09-28T07:47:03.158495671Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud failed: Get \"https://api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud:443/healthz?verbose\": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-rxbdcb" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc001090b40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-rxbdcb: failed waiting for cluster="cp-ystream-upgrade-4-23-rxbdcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-jvcp8dmnbrt7" to finish creating: GET https://rp.j4259200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "name": "1e7e2861-4689-447d-b71c-e2cfaeaee7cf",
      "status": "Failed",
      "startTime": "2026-09-28T07:17:56.829077997Z",
      "endTime": "2026-09-28T07:47:03.158495671Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud failed: Get \"https://api-int.t0n9t0u4r4r5j6y.biea.j4259200.hcp.osadev.cloud:443/healthz?verbose\": context deadline exceeded (Client.Timeout exceeded while awaiting headers); hosted cluster degraded: UnavailableReplicas: redhat-operators-catalog deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104457246334259200
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f0f2c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p82fzl2z5vk5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022",
      "name": "03f41a1e-bd33-4df1-9f47-56824bb74022",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:14.362679822Z",
      "endTime": "2026-09-28T05:24:15.510395217Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f0f2c0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p82fzl2z5vk5" to finish creating: GET https://rp.j4374016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/03f41a1e-bd33-4df1-9f47-56824bb74022",
      "name": "03f41a1e-bd33-4df1-9f47-56824bb74022",
      "status": "Failed",
      "startTime": "2026-09-28T04:55:14.362679822Z",
      "endTime": "2026-09-28T05:24:15.510395217Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingForAvailable: Waiting for Kube APIServer deployment to become available; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 3 unavailable replicas, router deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104421180684374016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, cluster-image-registry-operator, community-operators-catalog, hosted-cluster-config-operator, kube-apiserver, openshift-apiserver, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-c89c6r" with version 5.0.0-rc.3 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b10480>: 
    failed to create HCP cluster cluster-zstream-5-0-c89c6r: failed waiting for cluster="cluster-zstream-5-0-c89c6r" in resourcegroup="rg-zstream-upgrade-5-0-6kc9p4vcvjqq" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "name": "37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:45.206355596Z",
      "endTime": "2026-09-28T13:22:51.172490782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 26m23s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-c89c6r" with version 5.0.0-rc.3 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b10480>: 
    failed to create HCP cluster cluster-zstream-5-0-c89c6r: failed waiting for cluster="cluster-zstream-5-0-c89c6r" in resourcegroup="rg-zstream-upgrade-5-0-6kc9p4vcvjqq" to finish creating: GET https://rp.j8288256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "name": "37f813f1-0df9-4583-beb9-56f83fee5c0a",
      "status": "Failed",
      "startTime": "2026-09-28T12:53:45.206355596Z",
      "endTime": "2026-09-28T13:22:51.172490782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.3 is Partial (want Completed), started 26m23s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog, community-operators-catalog, kube-apiserver, redhat-marketplace-catalog, redhat-operators-catalog]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bf61a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lhz84lf5nw65" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119",
      "name": "045ddd50-d43d-42a0-bcda-90888c378119",
      "status": "Failed",
      "startTime": "2026-09-28T19:11:03.945176993Z",
      "endTime": "2026-09-28T19:40:05.721943753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m29s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bf61a0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lhz84lf5nw65" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/045ddd50-d43d-42a0-bcda-90888c378119",
      "name": "045ddd50-d43d-42a0-bcda-90888c378119",
      "status": "Failed",
      "startTime": "2026-09-28T19:11:03.945176993Z",
      "endTime": "2026-09-28T19:40:05.721943753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 23m29s ago; hosted cluster degraded: UnavailableReplicas: [certified-operators-catalog deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, kube-controller-manager, openshift-apiserver, packageserver, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6mht66l8hhvp/cluster-candidate-4-22-hxvn25 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00070f360>: 
    failed to create HCP cluster cluster-candidate-4-22-hxvn25: failed waiting for cluster="cluster-candidate-4-22-hxvn25" in resourcegroup="rg-candidate-4-22-6mht66l8hhvp" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "name": "889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:32.906905274Z",
      "endTime": "2026-09-28T19:39:35.528660951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-6mht66l8hhvp/cluster-candidate-4-22-hxvn25 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00070f360>: 
    failed to create HCP cluster cluster-candidate-4-22-hxvn25: failed waiting for cluster="cluster-candidate-4-22-hxvn25" in resourcegroup="rg-candidate-4-22-6mht66l8hhvp" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "name": "889a73a1-f8c7-4ed8-aad9-e3dfdb686168",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:32.906905274Z",
      "endTime": "2026-09-28T19:39:35.528660951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 23m1s ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc00060c440>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-br7vgttbd9zx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "name": "15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:44.986989982Z",
      "endTime": "2026-09-28T19:39:45.910063992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 24m39s ago; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:237]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc00060c440>: 
    failed waiting for cluster="acr-pull-create" in resourcegroup="acr-pull-create-br7vgttbd9zx" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "name": "15a2a59b-feef-488b-a5fa-5bcdbdb54fbf",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:44.986989982Z",
      "endTime": "2026-09-28T19:39:45.910063992Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 24m39s ago; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailab...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, openshift-apiserver, packageserver, redhat-operators-catalog, router]; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4rl6n98blqls/cluster-candidate-5-0-9zqn4p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f22920>: 
    failed to create HCP cluster cluster-candidate-5-0-9zqn4p: failed waiting for cluster="cluster-candidate-5-0-9zqn4p" in resourcegroup="rg-candidate-5-0-4rl6n98blqls" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927",
      "name": "b919a89b-ddbd-409e-ae0b-825760357927",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:07.120894243Z",
      "endTime": "2026-09-28T19:39:16.873419842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4rl6n98blqls/cluster-candidate-5-0-9zqn4p should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f22920>: 
    failed to create HCP cluster cluster-candidate-5-0-9zqn4p: failed waiting for cluster="cluster-candidate-5-0-9zqn4p" in resourcegroup="rg-candidate-5-0-4rl6n98blqls" to finish creating: GET https://rp.j0189952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b919a89b-ddbd-409e-ae0b-825760357927",
      "name": "b919a89b-ddbd-409e-ae0b-825760357927",
      "status": "Failed",
      "startTime": "2026-09-28T19:10:07.120894243Z",
      "endTime": "2026-09-28T19:39:16.873419842Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroup...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/<uuid>' at line '<line>' and column '<column>'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'; provider Microsoft.Authorization
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ca2e60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-z4h7jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-z4h7jf/operationStatuses/08584110018702456355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name/providers/Microsoft.Authorization/roleAssignments/ce4c794c-50e9-565e-99cc-b7dd7e08c5a9' at line '1' and column '29054'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 29054,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/e5697265-1ef6-54a3-8e48-b96178ae0611' at line '1' and column '30683'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 30683,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ca2e60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-z4h7jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-z4h7jf/operationStatuses/08584110018702456355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name/providers/Microsoft.Authorization/roleAssignments/ce4c794c-50e9-565e-99cc-b7dd7e08c5a9' at line '1' and column '29054'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 29054,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          },\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2wjjfp5h47hw/providers/Microsoft.Network/virtualNetworks/customer-vnet-name/providers/Microsoft.Authorization/roleAssignments/e5697265-1ef6-54a3-8e48-b96178ae0611' at line '1' and column '30683'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 30683,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71042104562485683032064
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRUpdateHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc002725230>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w8pbb4pwwmm7" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-w8pbb4pwwmm7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc002725230>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w8pbb4pwwmm7" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-w8pbb4pwwmm7, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet <namespace>/global-pull-secret-syncer has no desired pods scheduled
full failure pattern:
VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet <namespace>/global-pull-secret-syncer has no desired pods scheduled
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:183]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0xc000fb0020>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet kube-system/global-pull-secret-syncer has no desired pods scheduled
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:183]: failed to wait for global-pull-secret-syncer DaemonSet to be ready on nodepool nodes
Unexpected error:
    <*fmt.wrapError | 0xc000fb0020>: 
    VerifyDaemonSetReady(kube-system/global-pull-secret-syncer) timed out after 10m0.005s: DaemonSet kube-system/global-pull-secret-syncer has no desired pods scheduled
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71242104613528387194880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-...
full failure pattern:
VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullnsjbs8pwqtmk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00101eb60>: 
    VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e8a1ddc1-f3e1-4c99-ae14-bbe272e0b702; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d311f24-e917-4750-a06e-94b359669923
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:243]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullnsjbs8pwqtmk.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0xc00101eb60>: 
    VerifyImagePulled(acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal) timed out after 10m0.008s: expected image from acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
      - pod acr-pull-test container acr-pull-test: ImagePullBackOff - Back-off pulling image "acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://acrpullnsjbs8pwqtmk.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e8a1ddc1-f3e1-4c99-ae14-bbe272e0b702; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d311f24-e917-4750-a06e-94b359669923
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71812104634571122282496
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
full failure pattern:
VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001664240>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:140]: failed to verify cluster health or IngressController scope for cluster "private-ingress"
Unexpected error:
    <*errors.joinError | 0xc001664240>: 
    VerifyIngressControllerScope(Internal) failed: IngressController loadBalancer config is nil
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72312104701088555864064
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] KubeNodeUnreachable fired
full failure pattern:
alert [svc] KubeNodeUnreachable fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:21:03Z
  Ended:       2026-09-28T13:27:01Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="prometheus", node="aks-userswft3-30800232-vmss000000", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft3-30800232-vmss000000 is unreachable and some workloads may be rescheduled.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeNodeUnreachable does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104539007848288256
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
full failure pattern:
failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.14%1 of 707 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-b6k7qp"
Unexpected error:
    <*fmt.wrapError | 0xc000722f40>: 
    failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-b6k7qp"
Unexpected error:
    <*fmt.wrapError | 0xc000722f40>: 
    failed to create route: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
alert [svc] FrontendPathMedianLatency fired
full failure pattern:
alert [svc] FrontendPathMedianLatency fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 8 days
alert15421.78%154 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 8 daysShow trend detailsSep 21: 20 · Sep 22: 21 · Sep 23: 22 · Sep 24: 24 · Sep 25: 30 · Sep 26: 5 · Sep 27: 6 · Sep 28: 46none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T20:22:39Z
  Ended:       2026-09-28T20:28:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9312256-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9312256-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:18:56Z
  Ended:       2026-09-28T19:51:55Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0189952-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0189952-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T18:59:41Z
  Ended:       2026-09-28T19:05:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3028352-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3028352-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire154
Affected runs (154)
Date (UTC)Associated PRProw job
PR #69222104648687849312256
PR #71782104639037800189952
PR #70532104626712733028352
PR #69682104623441356263424
PR #72272104620444073070592
PR #71042104605657771544576
PR #70202104605132682432512
PR #72242104593036766875648
PR #71702104592047892598784
PR #70682104592038186979328
PR #71422104591950119178240
PR #69222104588552477085696
PR #72142104587545261117440
PR #71432104586688620662784
PR #72022104576830420815872
PR #69002104576709901684736
PR #72252104564648333283328
PR #70542104561257146224640
PR #72212104551522074365952
PR #69222104551505146155008
PR #71782104539007848288256
PR #71822104529106451304448
PR #72172104526339443789824
PR #72182104519303637241856
PR #71572104500979540955136
PR #68212104500972830068736
PR #72022104500902567088128
PR #72032104500868232515584
PR #69002104485944341041152
PR #62262104484432940371968
PR #71582104483794588274688
PR #72162104472130388234240
PR #72142104466453632651264
PR #71782104457246334259200
PR #72022104441054601154560
PR #72032104441035131195392
PR #68212104440974754189312
PR #70682104440930432978944
PR #68692104433568867422208
PR #62262104429033809776640
PR #71782104421180684374016
PR #71782104380048239759360
PR #70682104364976289353728
PR #68212104364960745263104
PR #71572104364925383086080
PR #72032104363449923407872
PR #71992104246785001132032
PR #71982104239747965652992
PR #71742104109345670172672
PR #71872104071407439712256
PR #71742104071277122686976
PR #71942104070538413477888
PR #71782103981635216084992
PR #71572103881469456814080
PR #71642103881459847663616
PR #68212103881428969197568
PR #70682103881405791473664
PR #71682103549055647879168
PR #71642103547560068124672
PR #71652103519763601821696
PR #69932103519758426050560
PR #70542103519728415805440
PR #70682103519715358937088
PR #68212103519651102199808
PR #70532103518634033811456
PR #71462103492665126424576
PR #71572103490540099080192
PR #65772103487715302445056
PR #71552103462425171857408
PR #69702103461761565855744
PR #71232103460627115675648
PR #65772103451563560275968
PR #71572103450358956167168
PR #71652103447011792523264
PR #70032103424207261011968
PR #70682103423928134275072
PR #69932103423855413432320
PR #71652103413943887204352
PR #66882103412592713142272
PR #70712103411013251502080
PR #71572103408234294415360
PR #70742103397005047894016
PR #71502103318253316083712
PR #69932103318216586563584
PR #70682103318217182154752
PR #71232103318164392644608
PR #71182103304771547959296
PR #69692103269477926834176
PR #71452103255589084729344
PR #71352103233926699945984
PR #70092103233919481548800
PR #70682103233417079427072
PR #70532103218829420138496
PR #71352103192892754366464
PR #70682103192838828199936
PR #71172103170613366493184
PR #71232103161541376348160
PR #70682103155003655983104
PR #69702103154812324417536
PR #70452103146310981914624
PR #71352103136210296770560
PR #71312103110260871925760
PR #67462103109521260941312
PR #69172103055129497309184
PR #71122103055091593383936
PR #71262103010871180529664
PR #55622102989394158817280
PR #62262102931134991044608
PR #71152102921426674651136
PR #71192102915118240108544
PR #71122102913782756937728
PR #69202102855242784706560
PR #69152102848491079340032
PR #69702102829281724862464
PR #71172102811735164456960
PR #70902102800084281331712
PR #69152102769175150727168
PR #70542102759065523326976
PR #71052102744695867183104
PR #69932102738633516650496
PR #71032102724367396900864
PR #70722102719452461142016
PR #64392102716371392532480
PR #70652102700766367059968
PR #62262102692523162996736
PR #69002102692072241762304
PR #66882102688892556677120
PR #70722102669953994854400
PR #69702102651461333684224
PR #70992102650881433407488
PR #70412102650243035172864
PR #69482102624604781547520
PR #69222102579128883482624
PR #70842102495173459054592
PR #66612102494708675645440
PR #70202102458797258706944
PR #48782102448371858935808
PR #68212102443148557946880
PR #69952102412746732605440
PR #68212102409636605857792
PR #70202102407137983467520
PR #70712102401634003849216
PR #70012102388132350529536
PR #65032102382852271247360
PR #61282102367426107674624
PR #67562102367380872105984
PR #70662102337463992193024
PR #70412102324260725854208
PR #69482102322606454607872
PR #69952102322580831604736
PR #69222102307266211352576
PR #70662102288344015704064
PR #69482102276691408719872
PR #69482102230390138736640
alert [svc] ServiceMemoryDrift fired
full failure pattern:
alert [svc] ServiceMemoryDrift fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
alert7710.89%77 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 17 · Sep 22: 31 · Sep 23: 18 · Sep 24: 28 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T18:08:30Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4417536-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2fbf2", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2fbf2 (namespace arobit) on cluster ci01-j4417536-svc has averaged 125.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:41:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1914624-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8vzwj", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-8vzwj (namespace arobit) on cluster ci01-j1914624-svc has averaged 126.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:49:20Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1279232-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5zn5h", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-5zn5h (namespace arobit) on cluster ci01-j1279232-svc has averaged 125.1% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire77
Affected runs (77)
Date (UTC)Associated PRProw job
PR #69702103154812324417536
PR #70452103146310981914624
PR #69172103145551821279232
PR #70532103139552272060416
PR #71352103136210296770560
PR #69152103114369016532992
PR #71312103110260871925760
PR #71322103106328183443456
PR #69172103099181529829376
PR #69222103099182641319936
PR #71232103089013568049152
PR #70092103078971544440832
PR #69222103055146278719488
PR #69172103055129497309184
PR #62262103055124225069056
PR #71272103055116536909824
PR #70452103034733737807872
PR #70092103026030053691392
PR #69952103024914473684992
PR #71262103010871180529664
PR #71052103001507350712320
PR #55622102989394158817280
PR #71122102961799233015808
PR #71152102921426674651136
PR #70542102916121228218368
PR #71232102914094729269248
PR #71122102913782756937728
PR #68462102912928209440768
PR #70202102883816627507200
PR #71232102872411731922944
PR #71192102871802198888448
PR #71082102862951168872448
PR #69202102855242784706560
PR #70542102840297615527936
PR #68212102840286248964096
PR #69702102829281724862464
PR #71172102811735164456960
PR #69252102800950539325440
PR #70542102795261179858944
PR #66872102764619155116032
PR #70442102751455361372160
PR #69932102738633516650496
PR #71032102724367396900864
PR #69482102669943756558336
PR #69702102651461333684224
PR #70412102650243035172864
PR #68212102534129747759104
PR #69932102490037739327488
PR #58462102463947360702464
PR #70202102458797258706944
PR #48782102448371858935808
PR #68212102443148557946880
PR #69002102424385783599104
PR #68892102424261724475392
PR #58462102421926960959488
PR #67562102412736506892288
PR #61282102412694500937728
PR #63642102408702790209536
PR #70722102399637812940800
PR #69222102397479768035328
PR #70012102388132350529536
PR #66882102385490215833600
PR #69482102367431941951488
PR #61282102367426107674624
PR #70702102359588845129728
PR #67522102358117298737152
PR #70662102337463992193024
PR #66882102336356570107904
PR #70412102324260725854208
PR #67562102322611546492928
PR #69952102322580831604736
PR #70412102283819666116608
PR #68912102236197437837312
PR #58462102231550941728768
PR #70602102210924612947968
PR #70592102191495971868672
PR #69452102186737458483200
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e689.62%68 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 91 · Sep 22: 91 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 14none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00172e090>: 
    resource group "complex-cilium-kv-k9knbpmhbg7v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k9knbpmhbg7v" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "name": "4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:43.521207454Z",
      "endTime": "2026-09-28T09:54:45.223507829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u15umjtvbfn7or4kh25ftp6mt8ubm still exists (deletion dispatched at 2026-09-28T09:30:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00172e090>: 
    resource group "complex-cilium-kv-k9knbpmhbg7v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-k9knbpmhbg7v" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "name": "4a3544af-5c77-496b-b18a-8cc097e8a79b",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:43.521207454Z",
      "endTime": "2026-09-28T09:54:45.223507829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u15umjtvbfn7or4kh25ftp6mt8ubm still exists (deletion dispatched at 2026-09-28T09:30:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002958120>: 
    resource group "rg-candidate-4-20-n6x2rfcpmr5t": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bkc8nt" in resourcegroup="rg-candidate-4-20-n6x2rfcpmr5t" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "name": "4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:44.717303002Z",
      "endTime": "2026-09-28T09:54:45.21106923Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ce2mv9aje3edhhstj5ffjhl0mpi still exists (deletion dispatched at 2026-09-28T09:31:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002958120>: 
    resource group "rg-candidate-4-20-n6x2rfcpmr5t": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-bkc8nt" in resourcegroup="rg-candidate-4-20-n6x2rfcpmr5t" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "name": "4d443cce-28d6-4421-9f77-bc36ad9031f7",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:44.717303002Z",
      "endTime": "2026-09-28T09:54:45.21106923Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0ce2mv9aje3edhhstj5ffjhl0mpi still exists (deletion dispatched at 2026-09-28T09:31:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001010288>: 
    resource group "clusternp128-cr6m72hbhthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cr6m72hbhthl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "name": "2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "status": "Failed",
      "startTime": "2026-09-28T09:26:14.153004619Z",
      "endTime": "2026-09-28T09:50:15.107483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ep16n5al4fc1ggfac6imb350l still exists (deletion dispatched at 2026-09-28T09:26:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001010288>: 
    resource group "clusternp128-cr6m72hbhthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-cr6m72hbhthl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "name": "2807e39f-5609-4e8d-9b63-6e0fb9dffa47",
      "status": "Failed",
      "startTime": "2026-09-28T09:26:14.153004619Z",
      "endTime": "2026-09-28T09:50:15.107483Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ep16n5al4fc1ggfac6imb350l still exists (deletion dispatched at 2026-09-28T09:26:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (30)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.2245
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest6
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.204
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.02
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
Affected runs (68)
Date (UTC)Associated PRProw job
PR #69002104485944341041152
PR #70412104360746694479872
PR #71762103505655406006272
PR #63592102530027982688256
PR #70842102495173459054592
PR #66612102494708675645440
PR #68212102494423198732288
PR #69932102490037739327488
PR #69252102479316322357248
PR #58462102463947360702464
PR #70202102458797258706944
PR #48782102448371858935808
PR #68212102443148557946880
PR #70672102428355495202816
PR #66612102425288674643968
PR #69002102424385783599104
PR #58462102421926960959488
PR #69952102412746732605440
PR #67562102412736506892288
PR #61282102412694500937728
PR #63642102408702790209536
PR #70542102408218671058944
PR #70202102407137983467520
PR #66622102400420944023552
PR #70722102399637812940800
PR #69222102397479768035328
PR #70012102388132350529536
PR #66882102385490215833600
PR #70672102382907266961408
PR #65032102382852271247360
PR #63642102371297248940032
PR #61282102367426107674624
PR #69952102367395485061120
PR #67562102367380872105984
PR #70702102359588845129728
PR #67522102358117298737152
PR #60922102353447767511040
PR #69222102352929464782848
PR #70672102340054952185856
PR #70662102337463992193024
PR #65032102337461471416320
PR #66882102336356570107904
PR #67562102322611546492928
PR #69482102322606454607872
PR #69952102322580831604736
PR #69222102307266211352576
PR #65032102295446469742592
PR #63642102294129609281536
PR #61282102288391285510144
PR #70662102288344015704064
PR #69952102286205365260288
PR #70412102283819666116608
PR #70532102276754965008384
PR #58462102276703479926784
PR #69482102276691408719872
PR #69452102272452842229760
PR #70602102246390443806720
PR #68912102236197437837312
PR #70532102231619598290944
PR #58462102231550941728768
PR #69452102231517219524608
PR #57242102230542270337024
PR #69482102230390138736640
PR #70602102210924612947968
PR #70612102203845328244736
PR #70592102191495971868672
PR #70532102186809273356288
PR #69452102186737458483200
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert334.67%33 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 21: 3 · Sep 22: 6 · Sep 23: 4 · Sep 24: 2 · Sep 25: 12 · Sep 26: 0 · Sep 27: 2 · Sep 28: 7none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-29T00:14:18Z
  Ended:       2026-09-29T00:26:21Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4504064-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T16:20:58Z
  Ended:       2026-09-28T16:27:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0662784-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T12:00:53Z
  Ended:       2026-09-28T12:14:51Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0514944-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T12:06:51Z
  Ended:       2026-09-28T12:18:52Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0514944-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire33
Affected runs (33)
Date (UTC)Associated PRProw job
PR #71242104709623964504064
PR #71432104586688620662784
PR #68842104529166530514944
PR #71822104529106451304448
PR #70682104500977863233536
PR #72022104500902567088128
PR #71572104364925383086080
PR #71972104201385095467008
PR #71852104002487378776064
PR #69932103519758426050560
PR #71432103507194874957824
PR #70742103489696960090112
PR #65772103487715302445056
PR #69202103485377787465728
PR #71232103460627115675648
PR #71572103450358956167168
PR #70682103423928134275072
PR #70092103419069259583488
PR #71352103318277487857664
PR #70682103318217182154752
PR #71412103305015505457152
PR #70452103146310981914624
PR #69172102965797063233536
PR #70202102883816627507200
PR #69152102848491079340032
PR #69002102782984254918656
PR #69222102579128883482624
PR #69002102424385783599104
PR #70792102409098145304576
PR #66882102385490215833600
PR #70662102337463992193024
PR #65032102295446469742592
PR #70602102246390443806720
alert [svc] FrontendPathLatency fired
full failure pattern:
alert [svc] FrontendPathLatency fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert223.11%22 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 4 · Sep 22: 13 · Sep 23: 3 · Sep 24: 5 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T10:21:49Z
  Ended:       2026-09-25T10:27:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6461184-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j6461184-svc).
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-25T10:32:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6461184-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationstatuses/{operationid}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/providers/microsoft.redhatopenshift/locations/{location}/hcpoperationstatuses/{operationid} exceeded 1 second over the past 30 minutes (cluster ci01-j6461184-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T17:31:32Z
  Ended:       2026-09-24T17:39:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1914624-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j1914624-svc).
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-24T12:40:11Z
  Ended:       2026-09-24T12:46:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4440832-svc", component="frontend", method="post", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential", severity="warning"
  Description: More than 5% of frontend requests for post /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential exceeded 1 second over the past 30 minutes (cluster ci01-j4440832-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire22
Affected runs (22)
Date (UTC)Associated PRProw job
PR #71052103404569546461184
PR #70452103146310981914624
PR #70092103078971544440832
PR #68842103015428132442112
PR #55622102989394158817280
PR #71152102921426674651136
PR #71102102850933644857344
PR #71152102812400645312512
PR #69222102579128883482624
PR #69222102534076756922368
PR #70892102530666527723520
PR #70842102495173459054592
PR #68212102443148557946880
PR #63642102371297248940032
PR #69482102367431941951488
PR #70662102337463992193024
PR #69482102322606454607872
PR #65162102321677739233280
PR #65032102295446469742592
PR #69482102276691408719872
PR #58462102231550941728768
PR #70532102186809273356288
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 8 days
alert182.55%18 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 8 daysShow trend detailsSep 21: 2 · Sep 22: 2 · Sep 23: 2 · Sep 24: 4 · Sep 25: 6 · Sep 26: 1 · Sep 27: 2 · Sep 28: 1none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T06:51:50Z
  Ended:       2026-09-28T06:57:49Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1154560-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3rl3f8tj7aka16mnasqurrb5ebbvn0-w7g1w7z1p7g5a8x", pod="kube-apiserver-8448d69f6f-jqnnb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3rl3f8tj7aka16mnasqurrb5ebbvn0-w7g1w7z1p7g5a8x/kube-apiserver-8448d69f6f-jqnnb has been in a non-ready state for longer than 5 minutes.
alert fired 13 time(s) (6 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:06:28Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T22:06:29Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-27T22:07:30Z
  Ended:       2026-09-27T22:48:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-27T22:07:30Z
  Ended:       2026-09-27T22:48:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-27T22:09:27Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-27T22:10:30Z
  Ended:       2026-09-27T22:49:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="etcd-0", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd-0 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T17:37:18Z
  Ended:       2026-09-27T17:43:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1132032-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t3g642vb3ovmhl4qiumvarv9qifaek4-c6e1u6h4n5k3a7j", pod="router-5584fc994b-5x9hp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t3g642vb3ovmhl4qiumvarv9qifaek4-c6e1u6h4n5k3a7j/router-5584fc994b-5x9hp has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire18
Affected runs (18)
Date (UTC)Associated PRProw job
PR #72022104441054601154560
PR #66882104319966160359424
PR #71992104246785001132032
PR #68212103881428969197568
PR #71572103490540099080192
PR #69702103461761565855744
PR #71232103460627115675648
PR #65772103451563560275968
PR #68212103396091608174592
PR #71572103346399469899776
PR #69702103203513864556544
PR #71412103191956178866176
PR #71302103111106493943808
PR #69172102919814468079616
PR #69152102848491079340032
PR #69222102624700403290112
PR #70642102534053163962368
PR #63642102371297248940032
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e182.55%18 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 0 · Sep 22: 2 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 4 · Sep 27: 4 · Sep 28: 7PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation18
Affected runs (18)
Date (UTC)Associated PRProw job
PR #69222104551505146155008
PR #72032104500868232515584
PR #71692104492976943140864
PR #68212104440974754189312
PR #71852104381379776090112
PR #71782104380048239759360
PR #72022104362505772994560
PR #66882104319966160359424
PR #71782104275998722756608
PR #69002104208646610096128
PR #71052104124987630686208
PR #71852103929841903472640
PR #71642103927280047755264
PR #71572103927255972450304
PR #71572103881469456814080
PR #65032103014767818969088
PR #68892102424261724475392
PR #68892102379019017129984
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 8 days
alert182.55%18 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 8 daysShow trend detailsSep 21: 1 · Sep 22: 3 · Sep 23: 2 · Sep 24: 4 · Sep 25: 1 · Sep 26: 2 · Sep 27: 1 · Sep 28: 5none
Full failure examples (3)
alert fired 79 time(s) (63 unknown, 16 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:22:59Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="router-5b97c8c479-td767", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/router-5b97c8c479-td767 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T19:23:58Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rbkaemrkauofk92ubess3a0bmc8a-v20261001", pod="router-688fdf9d74-7b7qb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rbkaemrkauofk92ubess3a0bmc8a-v20261001/router-688fdf9d74-7b7qb has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T19:25:04Z
  Ended:       2026-09-28T19:47:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster", pod="router-7856497d5-p9s72", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster/router-7856497d5-p9s72 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T19:25:04Z
  Ended:       2026-09-28T19:47:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster", pod="router-7856497d5-ptl86", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46s9suqafp0kib2eubo5b5ohtc2om9-agg-cluster/router-7856497d5-ptl86 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="ovnkube-control-plane-5fd96b4979-gbmgt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/ovnkube-control-plane-5fd96b4979-gbmgt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="ingress-operator-7d88f7b5dc-drxfd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/ingress-operator-7d88f7b5dc-drxfd has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-file-csi-driver-controller-854b478df8-v98lf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-file-csi-driver-controller-854b478df8-v98lf has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-disk-csi-driver-controller-5785d6645d-6xw4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-disk-csi-driver-controller-5785d6645d-6xw4f has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-disk-csi-driver-controller-5c84fd46c4-mpzts", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-disk-csi-driver-controller-5c84fd46c4-mpzts has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e", pod="azure-file-csi-driver-controller-559d66d9f8-v468g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re8cgt25tnc3e3r05ha4ola9p2m6-n3q8w5v1a0v4c7e/azure-file-csi-driver-controller-559d66d9f8-v468g has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:33:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-file-csi-driver-controller-9688c9fcd-vkkwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-file-csi-driver-controller-9688c9fcd-vkkwc has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-disk-csi-driver-controller-df8985bcf-xr5pq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-disk-csi-driver-controller-df8985bcf-xr5pq has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T19:25:59Z
  Ended:       2026-09-28T19:32:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="azure-file-csi-driver-controller-6f4b9c57cd-gpkcq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/azure-file-csi-driver-controller-6f4b9c57cd-gpkcq has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T19:29:02Z
  Ended:       2026-09-28T19:35:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6qhsdqg59jb18mtn75mdepd994t-p1e0l2c6g7e7x9p", pod="azure-disk-csi-driver-controller-d5b4b46ff-swrjk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6qhsdqg59jb18mtn75mdepd994t-p1e0l2c6g7e7x9p/azure-disk-csi-driver-controller-d5b4b46ff-swrjk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T19:29:02Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="ignition-server-6cf4c4cdbb-dcl2n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/ignition-server-6cf4c4cdbb-dcl2n has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="cluster-storage-operator-665d6949b6-6fw7x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/cluster-storage-operator-665d6949b6-6fw7x has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:37:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="ingress-operator-fb65c85-29jmc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/ingress-operator-fb65c85-29jmc has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="dns-operator-bfc958595-f2lvq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/dns-operator-bfc958595-f2lvq has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="kube-controller-manager-847bbfc79-gdtt6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/kube-controller-manager-847bbfc79-gdtt6 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-28T19:30:02Z
  Ended:       2026-09-28T19:36:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="cluster-network-operator-6995f687db-6cfj4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/cluster-network-operator-6995f687db-6cfj4 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-28T19:33:01Z
  Ended:       2026-09-28T19:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rb6d2v8ktanhkbba2l3aora65d7u-t1b3k0j9k6s4g3e", pod="kube-apiserver-5f69cc7c69-66mrs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rb6d2v8ktanhkbba2l3aora65d7u-t1b3k0j9k6s4g3e/kube-apiserver-5f69cc7c69-66mrs has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-network-operator-79b745b7-2lm4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-network-operator-79b745b7-2lm4f has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="kube-apiserver-c7844b576-98nm5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/kube-apiserver-c7844b576-98nm5 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-version-operator-767f75f6bd-czn9v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-version-operator-767f75f6bd-czn9v has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="network-node-identity-688597d98d-5nwfj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/network-node-identity-688597d98d-5nwfj has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="kube-apiserver-674ffb8fcf-w2vnq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/kube-apiserver-674ffb8fcf-w2vnq has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:39:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="csi-snapshot-controller-78d9d968f6-lktl7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/csi-snapshot-controller-78d9d968f6-lktl7 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="kube-apiserver-7f4977d86c-ggtv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/kube-apiserver-7f4977d86c-ggtv4 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="community-operators-catalog-77957f8659-vrvjj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/community-operators-catalog-77957f8659-vrvjj has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="multus-admission-controller-6659c5d88f-hgsf6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/multus-admission-controller-6659c5d88f-hgsf6 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="router-7964b968d8-s9n7t", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/router-7964b968d8-s9n7t has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="ignition-server-6cf4c4cdbb-xqn6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/ignition-server-6cf4c4cdbb-xqn6v has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="openshift-apiserver-79bff85c55-rmr6j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/openshift-apiserver-79bff85c55-rmr6j has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f", pod="openshift-apiserver-f4f5cc6bf-j27d5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9oavdebkcu57noiakdj4edk9fc6-c0q0y0c8u3m6c0f/openshift-apiserver-f4f5cc6bf-j27d5 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r619d7598ajep6f8mtkq1cmmun0r-g6f2d0d9e8y9l8w", pod="openshift-apiserver-855bf867d4-5dj69", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r619d7598ajep6f8mtkq1cmmun0r-g6f2d0d9e8y9l8w/openshift-apiserver-855bf867d4-5dj69 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-28T19:34:01Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="openshift-apiserver-548d95f77-j9rt4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/openshift-apiserver-548d95f77-j9rt4 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="kube-apiserver-8488cbccb8-9cntq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/kube-apiserver-8488cbccb8-9cntq has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="openshift-apiserver-59cf74f469-2pkbk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/openshift-apiserver-59cf74f469-2pkbk has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="cluster-image-registry-operator-5f9967bf67-m75d8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-image-registry-operator-5f9967bf67-m75d8 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="router-6966cb469f-jdx2g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/router-6966cb469f-jdx2g has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:46:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="kube-apiserver-6474ccdcd7-j5pkn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/kube-apiserver-6474ccdcd7-j5pkn has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a", pod="kube-apiserver-7877d486f4-dwd29", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46raguojej3hq4vd3uacec64tpmrts-g2q4v5m1s0e2c1a/kube-apiserver-7877d486f4-dwd29 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="network-node-identity-79f47f8db7-c6lsb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/network-node-identity-79f47f8db7-c6lsb has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i", pod="network-node-identity-6bf46b86d-r6lxm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r9cvp37q0olg8c158c363ap9mcq5-u6o0x8k8g4t9c8i/network-node-identity-6bf46b86d-r6lxm has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:48:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="router-d5fd8c684-npjp8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/router-d5fd8c684-npjp8 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="openshift-apiserver-7fccc5c996-2fqjh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/openshift-apiserver-7fccc5c996-2fqjh has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-09-28T19:35:03Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="certified-operators-catalog-56fcb687d5-x55km", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/certified-operators-catalog-56fcb687d5-x55km has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="openshift-apiserver-58689dc575-d9psz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/openshift-apiserver-58689dc575-d9psz has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="openshift-apiserver-fb96d7cd5-n8vx8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/openshift-apiserver-fb96d7cd5-n8vx8 has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="kube-apiserver-58866c8f4c-rssbm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/kube-apiserver-58866c8f4c-rssbm has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="openshift-apiserver-57597ccd79-kdsdf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/openshift-apiserver-57597ccd79-kdsdf has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l", pod="network-node-identity-6b45dfb8b5-gpsw6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3p7siuaq0ddmvgvscsi8di7b2ms-g5j1a0y1m9t0v0l/network-node-identity-6b45dfb8b5-gpsw6 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="router-5b97c8c479-87d9j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/router-5b97c8c479-87d9j has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create", pod="network-node-identity-cf9fb8b88-4lqsp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46rdha411s2s213ofkbr2v7t07eh2d-acr-pull-create/network-node-identity-cf9fb8b88-4lqsp has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-09-28T19:35:04Z
  Ended:       2026-09-28T19:45:59Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="kube-apiserver-6776ddb8b4-jtcfw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/kube-apiserver-6776ddb8b4-jtcfw has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-09-28T19:36:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-file-csi-driver-controller-68d9cdbd6c-bjqhs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-file-csi-driver-controller-68d9cdbd6c-bjqhs has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-disk-csi-driver-controller-688579f7b8-gcwdq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-disk-csi-driver-controller-688579f7b8-gcwdq has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="azure-disk-csi-driver-controller-778588d6c6-grn9w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/azure-disk-csi-driver-controller-778588d6c6-grn9w has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-09-28T19:36:02Z
  Ended:       2026-09-28T19:45:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="kube-controller-manager-5488d69fb6-p9ltk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/kube-controller-manager-5488d69fb6-p9ltk has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="kube-controller-manager-9864948dd-qzgnt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/kube-controller-manager-9864948dd-qzgnt has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="ovnkube-control-plane-58c8f9b6c5-45j6m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/ovnkube-control-plane-58c8f9b6c5-45j6m has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-09-28T19:37:01Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k", pod="network-node-identity-789b89cf4b-2v5sk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46r6c80n9jpgk27jtqr1mg2ert1grv-h9i2w9m0c5c0h0k/network-node-identity-789b89cf4b-2v5sk has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-09-28T19:38:02Z
  Ended:       2026-09-28T19:44:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0189952-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t46re3hoaml91vd90m68oiaekbg44as-e7l1h2p9d0o6f4j", pod="openshift-apiserver-5f8cc55fd4-bm995", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t46re3hoaml91vd90m68oiaekbg44as-e7l1h2p9d0o6f4j/openshift-apiserver-5f8cc55fd4-bm995 has been in a non-ready state for longer than 5 minutes.
alert fired 39 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qwlf6", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qwlf6 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-w28wh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-w28wh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-zxx8r", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-zxx8r has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-g76qj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-g76qj has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-d9qzz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-d9qzz has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-fpkv2", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-fpkv2 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-cwzx8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-cwzx8 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qm6c8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qm6c8 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-4996z", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-4996z has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-675f5", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-675f5 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-gd9k9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-gd9k9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-z58gp", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-z58gp has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-4j2hv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-4j2hv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-8n4fj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-8n4fj has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-8q8gs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-8q8gs has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-ddxpz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-ddxpz has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-9kqwf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-9kqwf has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-tpk28", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-tpk28 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-wpwxf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-wpwxf has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-c9pkv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-c9pkv has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-r2z8h", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-r2z8h has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-v6lst", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-v6lst has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-6nc22", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-6nc22 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-52zbx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-52zbx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-j8nbh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-j8nbh has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-f9lhx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-f9lhx has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dp559", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dp559 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-j4pwc", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-j4pwc has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-p9w7c", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-p9w7c has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-hqdp9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-hqdp9 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dgb65", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dgb65 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-rvk8h", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-rvk8h has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-mpfhk", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-mpfhk has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-qm2ts", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-qm2ts has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-27cpw", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-27cpw has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-mxsvz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-mxsvz has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-hgtjs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-hgtjs has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-c8pzf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-c8pzf has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-09-28T16:03:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0662784-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-7bcbfb996f-dvzkd", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-7bcbfb996f-dvzkd has been in a non-ready state for longer than 5 minutes.
alert fired 54 time(s) (38 unknown, 16 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:06:01Z
  Ended:       2026-09-28T13:29:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41bfp4l0n502nn624hggcdkif3fakm-l7r5c9r0t4i6i9i", pod="router-5f8dd5cdc8-2x7hv", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41bfp4l0n502nn624hggcdkif3fakm-l7r5c9r0t4i6i9i/router-5f8dd5cdc8-2x7hv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:13:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="cluster-network-operator-745b94c8d6-xgbrm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/cluster-network-operator-745b94c8d6-xgbrm has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:29:06Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster", pod="router-7f5958d584-h7qql", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster/router-7f5958d584-h7qql has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="openshift-apiserver-77dbbb84d4-g84qr", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/openshift-apiserver-77dbbb84d4-g84qr has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:32:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41aorgo7k4e5t9d3gl5386u8qrr1ok-b1g2t3h0x4w9w8x", pod="router-869d798bbd-2c6kt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41aorgo7k4e5t9d3gl5386u8qrr1ok-b1g2t3h0x4w9w8x/router-869d798bbd-2c6kt has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-28T13:07:01Z
  Ended:       2026-09-28T13:37:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster", pod="router-7f5958d584-xsp4j", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b7tbna4i6junuk36l82slrlj348n-cilium-cluster/router-7f5958d584-xsp4j has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-28T13:08:02Z
  Ended:       2026-09-28T13:35:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41bju2fkvns9a7dk4sqfvummalvm0f-t0e8p3i4e9p7l9u", pod="router-6449799f6-vwm42", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41bju2fkvns9a7dk4sqfvummalvm0f-t0e8p3i4e9p7l9u/router-6449799f6-vwm42 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-28T13:08:02Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="ignition-server-proxy-67ffb7567d-ndltn", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/ignition-server-proxy-67ffb7567d-ndltn has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-28T13:08:03Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41an05susenrnl0dfs40rcas06ch4c-f7z5i4t9g6o0n5p", pod="ignition-server-proxy-8699545598-fvwm5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41an05susenrnl0dfs40rcas06ch4c-f7z5i4t9g6o0n5p/ignition-server-proxy-8699545598-fvwm5 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-28T13:08:03Z
  Ended:       2026-09-28T13:26:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-controller-manager-867d6bcbc8-7zqfz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-controller-manager-867d6bcbc8-7zqfz has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-28T13:16:02Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="openshift-apiserver-db5f58d87-n2wk7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/openshift-apiserver-db5f58d87-n2wk7 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="kube-apiserver-85d5846f96-gpf28", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/kube-apiserver-85d5846f96-gpf28 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", pod="openshift-apiserver-85677f65c4-wrvhg", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics/openshift-apiserver-85677f65c4-wrvhg has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="network-node-identity-7549b9758b-4wb6s", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/network-node-identity-7549b9758b-4wb6s has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="openshift-apiserver-77dbbb84d4-29r9g", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/openshift-apiserver-77dbbb84d4-29r9g has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-apiserver-5c98b4447b-wvxv8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-apiserver-5c98b4447b-wvxv8 has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="kube-apiserver-745ff9b4db-srgc7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/kube-apiserver-745ff9b4db-srgc7 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="openshift-apiserver-84fcf4bc4d-mcmsx", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/openshift-apiserver-84fcf4bc4d-mcmsx has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="openshift-apiserver-b6b777476-w4j8h", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/openshift-apiserver-b6b777476-w4j8h has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="network-node-identity-57484c8884-trf2c", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/network-node-identity-57484c8884-trf2c has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="openshift-apiserver-5c68654d4b-dbjjt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/openshift-apiserver-5c68654d4b-dbjjt has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s", pod="openshift-apiserver-68c54f7686-xjxd2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41b29n97p684fi4p2rqp4ah5l33p66-l4y1n8h6k2x5m2s/openshift-apiserver-68c54f7686-xjxd2 has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress", pod="kube-apiserver-86c59c88cb-qrxm8", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41agsvgq9e8lrm448e04up5qhj06ch-private-ingress/kube-apiserver-86c59c88cb-qrxm8 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="kube-controller-manager-867d6bcbc8-bpsb5", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/kube-controller-manager-867d6bcbc8-bpsb5 has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="network-node-identity-65ff54759c-92pcp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/network-node-identity-65ff54759c-92pcp has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster", pod="network-node-identity-594c4dccf4-lm2n6", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ajnib77oktimcbmm0mfdc1ps458f-ea-cluster/network-node-identity-594c4dccf4-lm2n6 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics", pod="kube-apiserver-b8d4f754f-dqwb6", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anmrr6qaria4givc77cctbvucc8g-ksm-hcp-metrics/kube-apiserver-b8d4f754f-dqwb6 has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="kube-apiserver-957c77ffc-rg8wb", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/kube-apiserver-957c77ffc-rg8wb has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="kube-apiserver-5c9ff4f96c-n5vjw", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/kube-apiserver-5c9ff4f96c-n5vjw has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u", pod="openshift-apiserver-554784946b-mjxzz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41ba7k8pr9s4gom607lnh4dgn28ueo-g7e6w9u8n9t7m6u/openshift-apiserver-554784946b-mjxzz has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-09-28T13:17:07Z
  Ended:       2026-09-28T13:26:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j", pod="kube-apiserver-65f96cf6cc-d9g85", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am7qq1p8u0u2h4skvlto6j0jvekl-p5f5c9w6j9u4m2j/kube-apiserver-65f96cf6cc-d9g85 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-09-28T13:19:08Z
  Ended:       2026-09-28T13:26:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41anli60mk7obvb9qscekiln1r7nlh-z0b5k9r4h8u4l9q", pod="openshift-apiserver-5f4c5b7f7c-l5gpc", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41anli60mk7obvb9qscekiln1r7nlh-z0b5k9r4h8u4l9q/openshift-apiserver-5f4c5b7f7c-l5gpc has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-09-28T13:20:08Z
  Ended:       2026-09-28T13:26:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d", pod="multus-admission-controller-78665bcc7c-5f2bh", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am8gqalmfue5fmfh8v5u9ds73ppi-h6x4e1n8s3v5m1d/multus-admission-controller-78665bcc7c-5f2bh has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="csi-snapshot-controller-operator-74759598d-8dhlz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/csi-snapshot-controller-operator-74759598d-8dhlz has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="cluster-storage-operator-55cc4d6675-r7fxd", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/cluster-storage-operator-55cc4d6675-r7fxd has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="cluster-network-operator-5dc56d94dc-rmrjd", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/cluster-network-operator-5dc56d94dc-rmrjd has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:02Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="dns-operator-6d45bd4dc-c5h9t", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/dns-operator-6d45bd4dc-c5h9t has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-09-28T13:31:01Z
  Ended:       2026-09-28T13:41:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p", pod="ingress-operator-6c4557f7b9-x9crg", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t41am6903s65jni7s8tbqnvdtrqha12-n7t6b4k4k5z4g3p/ingress-operator-6c4557f7b9-x9crg has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire18
Affected runs (18)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71432104586688620662784
PR #71782104539007848288256
PR #71782104457246334259200
PR #71782104421180684374016
PR #71782104196521745453056
PR #71782103981635216084992
PR #71782103929857506283520
PR #71482103305281587908608
PR #68202103161107324604416
PR #71172103146302601695232
PR #65032103014767818969088
PR #70902102921738877669376
PR #68692102881520946515968
PR #68202102564185580769280
PR #68202102534072453566464
PR #68202102503516554661888
PR #68202102471925983875072
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e172.40%17 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 3 · Sep 22: 8 · Sep 23: 7 · Sep 24: 2 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000721c20>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8nk2s8" in resourcegroup="rg-np-noedge-8nk2s8-pbf4w4lpxflj" to finish creating: GET https://rp.j9067520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0",
      "name": "bf579354-82a6-4511-b470-a74098b9d7f0",
      "status": "Failed",
      "startTime": "2026-09-24T13:38:51.627589072Z",
      "endTime": "2026-09-24T13:57:59.764961372Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000721c20>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8nk2s8" in resourcegroup="rg-np-noedge-8nk2s8-pbf4w4lpxflj" to finish creating: GET https://rp.j9067520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf579354-82a6-4511-b470-a74098b9d7f0",
      "name": "bf579354-82a6-4511-b470-a74098b9d7f0",
      "status": "Failed",
      "startTime": "2026-09-24T13:38:51.627589072Z",
      "endTime": "2026-09-24T13:57:59.764961372Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00100c160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-wjm85h57dqlz" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921",
      "name": "deca3434-569c-4feb-abcd-1a0dae06f921",
      "status": "Failed",
      "startTime": "2026-09-24T08:38:16.231035723Z",
      "endTime": "2026-09-24T08:57:18.997473969Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00100c160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-wjm85h57dqlz" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/deca3434-569c-4feb-abcd-1a0dae06f921",
      "name": "deca3434-569c-4feb-abcd-1a0dae06f921",
      "status": "Failed",
      "startTime": "2026-09-24T08:38:16.231035723Z",
      "endTime": "2026-09-24T08:57:18.997473969Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc000a9f0c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-94qwsncfglvx" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "name": "9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "status": "Failed",
      "startTime": "2026-09-23T23:07:02.752849186Z",
      "endTime": "2026-09-23T23:26:11.541839688Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc000a9f0c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-94qwsncfglvx" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "name": "9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "status": "Failed",
      "startTime": "2026-09-23T23:07:02.752849186Z",
      "endTime": "2026-09-23T23:26:11.541839688Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (13)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (17)
Date (UTC)Associated PRProw job
PR #70652103098516569067520
PR #69952103024914473684992
PR #70202102883816627507200
PR #71102102850933644857344
PR #71082102818882191364096
PR #71142102800348405043200
PR #71062102751439267827712
PR #64392102750533910532096
PR #69932102738633516650496
PR #70532102506260145377280
PR #70712102401634003849216
PR #67562102367380872105984
PR #69932102366747628670976
PR #61282102322643775524864
PR #69952102286205365260288
PR #70412102283819666116608
PR #70602102246390443806720
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
e2e141.98%14 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 2 · Sep 24: 1 · Sep 25: 6 · Sep 26: 1 · Sep 27: 1 · Sep 28: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc001512f00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-5f68mkqs6vtp" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-5f68mkqs6vtp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc001512f00>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-5f68mkqs6vtp" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-5f68mkqs6vtp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc0033b6750>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-nh4vk2wgfp5z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-nh4vk2wgfp5z, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc0033b6750>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-nh4vk2wgfp5z" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-nh4vk2wgfp5z, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc003167da0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kg9lzmtgpr42" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kg9lzmtgpr42, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:209]: failed to enable autoscaling on node pool np-one-node
Unexpected error:
    <*fmt.wrapErrors | 0xc003167da0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-kg9lzmtgpr42" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-kg9lzmtgpr42, error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints4
Affected runs (14)
Date (UTC)Associated PRProw job
PR #71412104641856926126080
PR #72122104397716221071360
PR #71572104364925383086080
PR #71932104123485520400384
PR #71642103927280047755264
PR #71572103534322014228480
PR #71422103489841365782528
PR #71232103460627115675648
PR #70092103457038066520064
PR #68212103396091608174592
PR #71412103305015505457152
PR #71352103192892754366464
PR #63642102773722690621440
PR #70902102570502110318592
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 8 days
alert121.70%12 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 8 daysShow trend detailsSep 21: 4 · Sep 22: 3 · Sep 23: 2 · Sep 24: 1 · Sep 25: 3 · Sep 26: 1 · Sep 27: 1 · Sep 28: 1none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T21:44:52Z
  Ended:       2026-09-28T22:03:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T21:44:52Z
  Ended:       2026-09-28T22:00:53Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 12 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f3mdjcvo3r94pnej4b7fb4gsor4-basic-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6fa9chvq085h1gvcth6so4h4lfhv-k2s0j1o5g8h6x7l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6irrha761ifugv0q9ve23paki69c-ephemeral-disk/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-27T22:17:30Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6f5fj8gfr8k5odma9em9tq6i4f3j-f0z9g5b5n6v8h2l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-27T22:19:32Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-27T22:19:32Z
  Ended:       2026-09-27T22:44:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6j7re0275kpgl4hqp81isfr27n57-r2q8i7j0v0z2u1y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-27T22:20:28Z
  Ended:       2026-09-27T22:48:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-27T22:20:28Z
  Ended:       2026-09-27T22:44:27Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0359424-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.221:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0359424/providers/microsoft.monitor/accounts/hcps-j0359424", namespace="ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8zcr9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t3k6kln64qe3bnhq9dg58qvhurq1ran-u8r4i9k8p5r5w8l/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:06:56Z
  Ended:       2026-09-27T00:37:55Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7blf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-27T00:06:56Z
  Ended:       2026-09-27T00:36:52Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7blf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire12
Affected runs (12)
Date (UTC)Associated PRProw job
PR #72292104669678952517632
PR #66882104319966160359424
PR #71782103981635216084992
PR #71682103549055647879168
PR #71742103523744113758208
PR #70542103470957527044096
PR #69172102919814468079616
PR #63642102773722690621440
PR #70772102689595517833216
PR #70642102534053163962368
PR #65032102428363070115840
PR #65162102321677739233280
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert111.56%11 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 2 · Sep 22: 3 · Sep 23: 0 · Sep 24: 0 · Sep 25: 2 · Sep 26: 1 · Sep 27: 0 · Sep 28: 5none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T21:47:52Z
  Ended:       2026-09-28T22:02:50Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2517632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2517632/providers/microsoft.monitor/accounts/hcps-j2517632", namespace="ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2xkdh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t48kv9d505es3p8ivbsahgh3k9orrao-z6g9i6m9w7w8g8g/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r598rg9utsesqokqgsh5is4bsff4-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46ri3b6cnjabfmbna1k3ic8p591bg1-n6n8b8u3w4n1g4l/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T20:21:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/hcps-j0189952", namespace="ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t46r3spp95klv4no3blpdjir2phpkgs-v4e7b2k4r8i1q2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:12:48Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z8g6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:15:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:15:54Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:16:53Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2d7tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:17:49Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z8g6d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire11
Affected runs (11)
Date (UTC)Associated PRProw job
PR #72292104669678952517632
PR #71782104639037800189952
PR #69002104485944341041152
PR #71782104457246334259200
PR #71782104421180684374016
PR #71572103927255972450304
PR #71742103523744113758208
PR #71572103490540099080192
PR #70642102534053163962368
PR #68892102424261724475392
PR #65162102321677739233280
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e111.56%11 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 0 · Sep 22: 4 · Sep 23: 1 · Sep 24: 4 · Sep 25: 0 · Sep 26: 0 · Sep 27: 2 · Sep 28: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000676bd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000676bd0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002177290>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.005s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002177290>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001ea9b70>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.003s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001ea9b70>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges11
Affected runs (11)
Date (UTC)Associated PRProw job
PR #71962104191454321577984
PR #71872104071407439712256
PR #63642103224244262408192
PR #70452103078389232439296
PR #71272103015078914691072
PR #71262103010871180529664
PR #70652102700766367059968
PR #58462102463947360702464
PR #58462102421926960959488
PR #63642102408702790209536
PR #70542102408218671058944
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert91.27%9 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 21: 3 · Sep 22: 2 · Sep 23: 2 · Sep 24: 1 · Sep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 2none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:45:04Z
  Ended:       2026-09-28T19:51:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="klusterlet-2t46re8cgt25tnc3e3r05ha4ola9p2m6", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t46re8cgt25tnc3e3r05ha4ola9p2m6/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/services-j4374016", namespace="klusterlet-2t3qakt150613r65up3s1cocvavjoksi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t3qakt150613r65up3s1cocvavjoksi/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T05:30:55Z
  Ended:       2026-09-28T05:36:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4374016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4374016/providers/microsoft.monitor/accounts/services-j4374016", namespace="klusterlet-2t3qakt150613r65up3s1cocvavjoksi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zzx2c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t3qakt150613r65up3s1cocvavjoksi/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:21:19Z
  Ended:       2026-09-27T00:27:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/services-j6084992", namespace="klusterlet-2t30mivbmcgkot8vhp8oa13ibv1vnaut", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7blf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t30mivbmcgkot8vhp8oa13ibv1vnaut/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire9
Affected runs (9)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71782104421180684374016
PR #71782103981635216084992
PR #71482103305281587908608
PR #65032103014767818969088
PR #68692102881520946515968
PR #71082102772525283938304
PR #70642102534053163962368
PR #65032102428363070115840
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e91.27%9 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 1 · Sep 22: 0 · Sep 23: 2 · Sep 24: 2 · Sep 25: 3 · Sep 26: 0 · Sep 27: 2 · Sep 28: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000f972a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-t7rmzspt8s2k" to finish creating: GET https://rp.j6124032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "name": "0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "status": "Failed",
      "startTime": "2026-09-27T20:26:25.730010902Z",
      "endTime": "2026-09-27T20:45:26.626992422Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k: Failed: interfaces failed to create or update. err: failed to get existing resource nodepool-delete-t7rmzspt8s2k--managed/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-t7rmzspt8s2k--managed/providers/Microsoft.Network/networkInterfaces/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' seconds after additiona... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000f972a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-t7rmzspt8s2k" to finish creating: GET https://rp.j6124032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "name": "0aabc56d-9f58-4448-ac0c-c46df31d92cc",
      "status": "Failed",
      "startTime": "2026-09-27T20:26:25.730010902Z",
      "endTime": "2026-09-27T20:45:26.626992422Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k: Failed: interfaces failed to create or update. err: failed to get existing resource nodepool-delete-t7rmzspt8s2k--managed/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-t7rmzspt8s2k--managed/providers/Microsoft.Network/networkInterfaces/q2q3v2b3m8b4m9x-np-delete-main-7cgrv-l7n6k-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '12eafc4e-f869-4b62-8198-816b7d6d0876' exceeded. Please try again after '10' seconds after additiona... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00089fe00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-25nvltvpwf84" to finish creating: GET https://rp.j5652992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "name": "6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "status": "Failed",
      "startTime": "2026-09-27T17:01:59.070822701Z",
      "endTime": "2026-09-27T17:21:03.436757642Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-25l2v-tmrtf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-25l2v-tmrtf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc00089fe00>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-25nvltvpwf84" to finish creating: GET https://rp.j5652992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "name": "6c359dd9-afd9-4a97-8e2f-ad9a26c6937d",
      "status": "Failed",
      "startTime": "2026-09-27T17:01:59.070822701Z",
      "endTime": "2026-09-27T17:21:03.436757642Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-25l2v-tmrtf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-25l2v-tmrtf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-25nvltvpwf84--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-25l2v-tmrtf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000adbee0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qqrdr5ldkfxf" to finish creating: GET https://rp.j2365184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "name": "e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "status": "Failed",
      "startTime": "2026-09-25T20:55:06.599822055Z",
      "endTime": "2026-09-25T21:14:14.361175634Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000adbee0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qqrdr5ldkfxf" to finish creating: GET https://rp.j2365184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "name": "e499c0b9-b430-40af-9ef1-8fccc02c39c5",
      "status": "Failed",
      "startTime": "2026-09-25T20:55:06.599822055Z",
      "endTime": "2026-09-25T21:14:14.361175634Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-qqrdr5ldkfxf--managed/providers/Microsoft.Compute/virtualMachines/j0r0n4d0n1i1i3m-np-delete-main-gskm4-2m8j2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool7
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
Affected runs (9)
Date (UTC)Associated PRProw job
PR #71832104290856126124032
PR #71982104239747965652992
PR #71722103574351302365184
PR #69932103519758426050560
PR #70032103424207261011968
PR #71352103233926699945984
PR #70092103192810386624512
PR #71172102811735164456960
PR #69932102738633516650496
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e81.13%8 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 21: 1 · Sep 22: 7 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000eb40f0>: 
    resource group "fips-enabled-bn5q2wwsf7rp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn5q2wwsf7rp" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "name": "3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:05.789845252Z",
      "endTime": "2026-09-28T09:54:14.850550256Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0hvkv6dq5otq5pkoot1lod893ptj still exists (deletion dispatched at 2026-09-28T09:30:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000eb40f0>: 
    resource group "fips-enabled-bn5q2wwsf7rp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-bn5q2wwsf7rp" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "name": "3a6f097a-b7cb-40aa-8dcb-c39bec4a86df",
      "status": "Failed",
      "startTime": "2026-09-28T09:30:05.789845252Z",
      "endTime": "2026-09-28T09:54:14.850550256Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0hvkv6dq5otq5pkoot1lod893ptj still exists (deletion dispatched at 2026-09-28T09:30:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001700288>: 
    resource group "private-keyvault-vfws486bsfgl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-vfws486bsfgl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "name": "5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "status": "Failed",
      "startTime": "2026-09-28T09:29:57.435847003Z",
      "endTime": "2026-09-28T09:54:04.986151784Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u18j4sj43hhdfhhr4o61io72f8235 still exists (deletion dispatched at 2026-09-28T09:29:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001700288>: 
    resource group "private-keyvault-vfws486bsfgl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-vfws486bsfgl" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "name": "5f4a43f8-8823-4442-9a9f-327ed4ef5fef",
      "status": "Failed",
      "startTime": "2026-09-28T09:29:57.435847003Z",
      "endTime": "2026-09-28T09:54:04.986151784Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u18j4sj43hhdfhhr4o61io72f8235 still exists (deletion dispatched at 2026-09-28T09:29:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00290a2b8>: 
    resource group "rg-candidate-5-0-8h76gpwqntfk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-5z7vd6" in resourcegroup="rg-candidate-5-0-8h76gpwqntfk" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "name": "cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "status": "Failed",
      "startTime": "2026-09-28T09:35:00.890511025Z",
      "endTime": "2026-09-28T09:59:04.96675702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ao372vngnfec0pqf548ff4sjj still exists (deletion dispatched at 2026-09-28T09:35:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00290a2b8>: 
    resource group "rg-candidate-5-0-8h76gpwqntfk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-5z7vd6" in resourcegroup="rg-candidate-5-0-8h76gpwqntfk" to finish deleting: GET https://rp.j1041152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "name": "cf79dcd4-21ea-4d96-a412-53a56f9ed792",
      "status": "Failed",
      "startTime": "2026-09-28T09:35:00.890511025Z",
      "endTime": "2026-09-28T09:59:04.96675702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t3u0n0ao372vngnfec0pqf548ff4sjj still exists (deletion dispatched at 2026-09-28T09:35:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (8)
Date (UTC)Associated PRProw job
PR #69002104485944341041152
PR #63592102530027982688256
PR #58462102463947360702464
PR #67562102412736506892288
PR #69482102367431941951488
PR #70312102329484735680512
PR #69452102231517219524608
PR #70602102210924612947968
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e81.13%8 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 0 · Sep 22: 2 · Sep 23: 1 · Sep 24: 1 · Sep 25: 2 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-x7z7b7"
Unexpected error:
    <*fmt.wrapError | 0xc000bab020>: 
    DNS for route host agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-x7z7b7"
Unexpected error:
    <*fmt.wrapError | 0xc000bab020>: 
    DNS for route host agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvf99.apps.aro.t2p4s2i1j4x6p8f.mtn4.j7422208.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000c3c580>: 
    DNS for route host agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xc000c3c580>: 
    DNS for route host agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqp78.apps.aro.v20261001.jzew.j9776640.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc002da48c0>: 
    DNS for route host agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc002da48c0>: 
    DNS for route host agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n4r5h.apps.aro.v20260901.fndt.j7290752.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (8)
Date (UTC)Associated PRProw job
PR #68692104433568867422208
PR #62262104429033809776640
PR #71042103584488377290752
PR #71422103555692374790144
PR #69152103012742733500416
PR #69692102853671329992704
PR #70532102276754965008384
PR #69482102276691408719872
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e70.99%7 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 0 · Sep 22: 1 · Sep 23: 2 · Sep 24: 2 · Sep 25: 4 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000990ae0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wf95j7vx69b5" to finish creating: GET https://rp.j5675648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2",
      "name": "00381033-aeb7-400a-bd07-e1bd8611cea2",
      "status": "Failed",
      "startTime": "2026-09-25T13:15:51.082389682Z",
      "endTime": "2026-09-25T13:45:00.023728639Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000990ae0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-wf95j7vx69b5" to finish creating: GET https://rp.j5675648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/00381033-aeb7-400a-bd07-e1bd8611cea2",
      "name": "00381033-aeb7-400a-bd07-e1bd8611cea2",
      "status": "Failed",
      "startTime": "2026-09-25T13:15:51.082389682Z",
      "endTime": "2026-09-25T13:45:00.023728639Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6ghnhlc94h8f/cluster-hs-pre-5wm67n should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010f25a0>: 
    failed to create HCP cluster cluster-hs-pre-5wm67n: failed waiting for cluster="cluster-hs-pre-5wm67n" in resourcegroup="rg-hs-pre--6ghnhlc94h8f" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911",
      "name": "46dab79f-2360-4680-914d-589d803cc911",
      "status": "Failed",
      "startTime": "2026-09-25T05:47:00.77129988Z",
      "endTime": "2026-09-25T06:16:03.97431147Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--6ghnhlc94h8f/cluster-hs-pre-5wm67n should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010f25a0>: 
    failed to create HCP cluster cluster-hs-pre-5wm67n: failed waiting for cluster="cluster-hs-pre-5wm67n" in resourcegroup="rg-hs-pre--6ghnhlc94h8f" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46dab79f-2360-4680-914d-589d803cc911",
      "name": "46dab79f-2360-4680-914d-589d803cc911",
      "status": "Failed",
      "startTime": "2026-09-25T05:47:00.77129988Z",
      "endTime": "2026-09-25T06:16:03.97431147Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m28s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c1aa00>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pgng9vz6j8w8" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327",
      "name": "a614d325-8937-4e4c-a91f-47b2564f0327",
      "status": "Failed",
      "startTime": "2026-09-25T05:48:11.218096649Z",
      "endTime": "2026-09-25T06:17:14.62086737Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m30s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c1aa00>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-pgng9vz6j8w8" to finish creating: GET https://rp.j9899776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a614d325-8937-4e4c-a91f-47b2564f0327",
      "name": "a614d325-8937-4e4c-a91f-47b2564f0327",
      "status": "Failed",
      "startTime": "2026-09-25T05:48:11.218096649Z",
      "endTime": "2026-09-25T06:17:14.62086737Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m30s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (7)
Date (UTC)Associated PRProw job
PR #71232103460627115675648
PR #71572103346399469899776
PR #69702103203513864556544
PR #71412103191956178866176
PR #69152102848491079340032
PR #70442102795270910644224
PR #63642102371297248940032
alert [svc] KubeDaemonSetRolloutStuck fired
full failure pattern:
alert [svc] KubeDaemonSetRolloutStuck fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
alert70.99%7 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 0 · Sep 22: 3 · Sep 23: 1 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (3)
alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T19:37:02Z
  Ended:       2026-09-28T19:45:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-28T19:39:13Z
  Ended:       2026-09-28T19:45:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0189952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.44:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0189952/providers/microsoft.monitor/accounts/services-j0189952", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xp2wg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-28T13:18:05Z
  Ended:       2026-09-28T13:31:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-28T13:18:05Z
  Ended:       2026-09-28T13:29:07Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8288256-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="azure-cns", endpoint="http", environment="ci01", instance="10.128.65.50:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8288256/providers/microsoft.monitor/accounts/services-j8288256", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-789nk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/azure-cns has not finished or progressed for at least 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:34:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.41:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4604416/providers/microsoft.monitor/accounts/services-j4604416", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-mmg7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-24T17:34:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4604416-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.41:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4604416/providers/microsoft.monitor/accounts/services-j4604416", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-mmg7d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #71782104639037800189952
PR #71782104539007848288256
PR #68202103161107324604416
PR #68202102564185580769280
PR #68202102534072453566464
PR #68202102503516554661888
PR #68202102471925983875072
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e60.85%6 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 4 · Sep 22: 3 · Sep 23: 2 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--nrk2cdj4tcn5/cluster-hs-pre-pcqks7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009409a0>: 
    failed to create HCP cluster cluster-hs-pre-pcqks7: failed waiting for cluster="cluster-hs-pre-pcqks7" in resourcegroup="rg-hs-pre--nrk2cdj4tcn5" to finish creating: GET https://rp.j7044096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea",
      "name": "e89da08a-794d-4885-a19f-3eb28678e2ea",
      "status": "Failed",
      "startTime": "2026-09-25T13:47:52.911662059Z",
      "endTime": "2026-09-25T14:16:59.018766333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--nrk2cdj4tcn5/cluster-hs-pre-pcqks7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009409a0>: 
    failed to create HCP cluster cluster-hs-pre-pcqks7: failed waiting for cluster="cluster-hs-pre-pcqks7" in resourcegroup="rg-hs-pre--nrk2cdj4tcn5" to finish creating: GET https://rp.j7044096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e89da08a-794d-4885-a19f-3eb28678e2ea",
      "name": "e89da08a-794d-4885-a19f-3eb28678e2ea",
      "status": "Failed",
      "startTime": "2026-09-25T13:47:52.911662059Z",
      "endTime": "2026-09-25T14:16:59.018766333Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0018bec90>: 
    GET https://rp.aro-hcp-dev-shard3-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e",
      "name": "54444a02-0c9c-4321-935a-b1997c03724e",
      "status": "Failed",
      "startTime": "2026-09-23T15:55:02.945988001Z",
      "endTime": "2026-09-23T16:24:05.016015435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0018bec90>: 
    GET https://rp.aro-hcp-dev-shard3-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e",
      "name": "54444a02-0c9c-4321-935a-b1997c03724e",
      "status": "Failed",
      "startTime": "2026-09-23T15:55:02.945988001Z",
      "endTime": "2026-09-23T16:24:05.016015435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001078700>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-tsmxlnchqjdn" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "name": "5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "status": "Failed",
      "startTime": "2026-09-23T10:04:12.923013072Z",
      "endTime": "2026-09-23T10:33:15.61163171Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001078700>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-tsmxlnchqjdn" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "name": "5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "status": "Failed",
      "startTime": "2026-09-23T10:04:12.923013072Z",
      "endTime": "2026-09-23T10:33:15.61163171Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (6)
Date (UTC)Associated PRProw job
PR #70542103470957527044096
PR #63642102773722690621440
PR #70772102689595517833216
PR #70642102534053163962368
PR #65032102428363070115840
PR #65162102321677739233280
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert60.85%6 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 21: 2 · Sep 22: 2 · Sep 23: 1 · Sep 24: 0 · Sep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 1none
Full failure examples (3)
alert fired 16 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="cloud-network-config-controller-58dfb59986-bsjtm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2b4fc680-6a59-448e-b315-f78694553d9a"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller-58dfb59986-bsjtm (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="cloud-network-config-controller-69985fdf59-fmd5d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="387b4461-aa28-4131-85f9-f2b77c4e2e0e"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/cloud-network-config-controller-69985fdf59-fmd5d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="cloud-network-config-controller-69985fdf59-fmd5d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="387b4461-aa28-4131-85f9-f2b77c4e2e0e"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/cloud-network-config-controller-69985fdf59-fmd5d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="csi-snapshot-controller-d4d6cd4f4-d2tqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c3f26a0c-8e66-4f75-bb2e-f2645071d798"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/csi-snapshot-controller-d4d6cd4f4-d2tqr (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:33:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e", pod="csi-snapshot-controller-d4d6cd4f4-d2tqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c3f26a0c-8e66-4f75-bb2e-f2645071d798"
  Description: Pod ocm-arohcpci01-2t3u0n0ao372vngnfec0pqf548ff4sjj-d9g9f6v4c3w8y9e/csi-snapshot-controller-d4d6cd4f4-d2tqr (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="csi-snapshot-controller-c9cf48cfd-vfz96", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e625ddc5-e22f-4d5c-bb2f-d4cb602de924"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/csi-snapshot-controller-c9cf48cfd-vfz96 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="cloud-network-config-controller-5c87577d75-pzm4d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f7312b5e-d9af-4928-8d9a-ea34f1380f1e"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller-5c87577d75-pzm4d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="cloud-network-config-controller-5b9c45d5bc-245tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e7400d-4c21-4a6d-a049-fcf41ebbe075"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller-5b9c45d5bc-245tg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="cloud-network-config-controller-58dfb59986-bsjtm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2b4fc680-6a59-448e-b315-f78694553d9a"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/cloud-network-config-controller-58dfb59986-bsjtm (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="csi-snapshot-controller-69d89876cf-6p78f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a54f46c-0195-4d5e-aae8-68c97f711f84"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/csi-snapshot-controller-69d89876cf-6p78f (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="cloud-network-config-controller-5c87577d75-pzm4d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f7312b5e-d9af-4928-8d9a-ea34f1380f1e"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/cloud-network-config-controller-5c87577d75-pzm4d (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.91:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y", pod="csi-snapshot-controller-69d89876cf-6p78f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a54f46c-0195-4d5e-aae8-68c97f711f84"
  Description: Pod ocm-arohcpci01-2t3u0lp2q6dsf21b8209r6h4nf0vedlp-e1u8w7m0a8d6a5y/csi-snapshot-controller-69d89876cf-6p78f (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="csi-snapshot-controller-7d6c8c6fbb-xmtc5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07ef49a6-b3c7-427b-ab6f-44204ea64710"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/csi-snapshot-controller-7d6c8c6fbb-xmtc5 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b", pod="csi-snapshot-controller-c9cf48cfd-vfz96", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e625ddc5-e22f-4d5c-bb2f-d4cb602de924"
  Description: Pod ocm-arohcpci01-2t3u0ffhfsef977jtdd70rv0ev42cvrg-k2l3d4g2t9f1a4b/csi-snapshot-controller-c9cf48cfd-vfz96 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="cloud-network-config-controller-5b9c45d5bc-245tg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e7400d-4c21-4a6d-a049-fcf41ebbe075"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/cloud-network-config-controller-5b9c45d5bc-245tg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-28T10:34:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1041152-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1041152/providers/microsoft.monitor/accounts/hcps-j1041152", namespace="ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n", pod="csi-snapshot-controller-7d6c8c6fbb-xmtc5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="07ef49a6-b3c7-427b-ab6f-44204ea64710"
  Description: Pod ocm-arohcpci01-2t3u0kbt737hlqiqbf1dq3vg54p17efo-n1w7s8z8q6s5h2n/csi-snapshot-controller-7d6c8c6fbb-xmtc5 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-27T00:09:50Z
  Ended:       2026-09-27T00:37:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bb068423-1854-49c8-a132-0d10a6392326"
  Description: Pod ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-27T00:09:50Z
  Ended:       2026-09-27T00:37:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6084992-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6084992/providers/microsoft.monitor/accounts/hcps-j6084992", namespace="ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bb068423-1854-49c8-a132-0d10a6392326"
  Description: Pod ocm-arohcpci01-2t30mivbmcgkot8vhp8oa13ibv1vnaut-private-ingress/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-25T19:26:00Z
  Ended:       2026-09-25T19:56:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7879168-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7879168/providers/microsoft.monitor/accounts/hcps-j7879168", namespace="ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fd1ced12-4634-450b-be94-fe821eb628d0"
  Description: Pod ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-25T19:26:00Z
  Ended:       2026-09-25T19:55:01Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j7879168-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.83:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7879168/providers/microsoft.monitor/accounts/hcps-j7879168", namespace="ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fd1ced12-4634-450b-be94-fe821eb628d0"
  Description: Pod ocm-arohcpci01-2t27f8egsdoeqjvu0j3vd58dlko2an9h-r5q1e3i5h8g3j3f/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #69002104485944341041152
PR #71782103981635216084992
PR #71682103549055647879168
PR #63642102773722690621440
PR #70642102534053163962368
PR #65032102428363070115840
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e60.85%6 of 707 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 0 · Sep 22: 2 · Sep 23: 2 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000693320>: 
    DNS for route host agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000693320>: 
    DNS for route host agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-swv95.apps.aro.u1w0b2p8i5n8f3h.2e4g.j0616448.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-bxncxk"
Unexpected error:
    <*fmt.wrapError | 0xc000b16880>: 
    DNS for route host agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-bxncxk"
Unexpected error:
    <*fmt.wrapError | 0xc000b16880>: 
    DNS for route host agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lvch8.apps.aro.n2m9k8u9s3b0p8r.wpls.aro-hcp-dev-shard2-slot-03.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000f1c4c0>: 
    DNS for route host agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000f1c4c0>: 
    DNS for route host agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (6)
Date (UTC)Associated PRProw job
PR #71062104551230880616448
PR #69152103012742733500416
PR #69002102782984254918656
PR #69202102762797665357824
PR #48782102448371858935808
PR #63642102294129609281536

Environment: INT

Failure patterns: 9 · Failures matched: 15 · Total runs: 30 · Runs affected: 11

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e310.00%3 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 4 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000599de0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-kcj4l5" in resourcegroup="rg-np-skip-minor-kcj4l5-45s8nhm4k78b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d",
      "name": "da564fd3-aea3-4a41-b171-3969372d023d",
      "status": "Failed",
      "startTime": "2026-09-25T16:39:04.653915252Z",
      "endTime": "2026-09-25T16:58:09.716908828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000599de0>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-kcj4l5" in resourcegroup="rg-np-skip-minor-kcj4l5-45s8nhm4k78b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da564fd3-aea3-4a41-b171-3969372d023d",
      "name": "da564fd3-aea3-4a41-b171-3969372d023d",
      "status": "Failed",
      "startTime": "2026-09-25T16:39:04.653915252Z",
      "endTime": "2026-09-25T16:58:09.716908828Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-kcj4l5-45s8nhm4k78b-np-sm-kcj4l5--managed/providers/Microsoft.Compute/virtualMachines/k7t8o1t6u7r7f0q-nps-420-422-lf9bj-qzzxg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00075fc80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-dnhw57q2zhsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "name": "9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "status": "Failed",
      "startTime": "2026-09-25T08:30:53.138668141Z",
      "endTime": "2026-09-25T08:49:53.966422643Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00075fc80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-dnhw57q2zhsr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "name": "9cb82bcd-1a84-400b-ad2e-8cf3dfa50902",
      "status": "Failed",
      "startTime": "2026-09-25T08:30:53.138668141Z",
      "endTime": "2026-09-25T08:49:53.966422643Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dnhw57q2zhsr--managed/providers/Microsoft.Compute/virtualMachines/q8f0o9t2z5z1x2r-np-1-9lssw-jg4gh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010c02c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2mw49n77wnb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "name": "e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "status": "Failed",
      "startTime": "2026-09-25T08:27:10.94071903Z",
      "endTime": "2026-09-25T08:46:13.959695236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc0010c02c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-2mw49n77wnb2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "name": "e57de6e4-350f-44ca-8989-0fa815b3fd2f",
      "status": "Failed",
      "startTime": "2026-09-25T08:27:10.94071903Z",
      "endTime": "2026-09-25T08:46:13.959695236Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-2mw49n77wnb2--managed/providers/Microsoft.Compute/virtualMachines/y5d4i4z6u4m1b4g-nodepool-128gib-j5q6z-kjlzl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2103500347635929088
n/a2103397522356572160
n/a2103303905172525056
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e26.67%2 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 1DEV, PROD, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb"
Unexpected error:
    <*fmt.wrapError | 0xc00024f2c0>: 
    failed to create NodePool np-fnv7lb: failed waiting for nodepool="np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb" in resourcegroup="rg-candidate-5-0-v6nvmt4rxzjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a",
      "name": "203ad465-1e1f-4337-9659-f5fe2972a72a",
      "status": "Failed",
      "startTime": "2026-09-28T21:17:06.138034027Z",
      "endTime": "2026-09-28T21:36:06.809868093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb"
Unexpected error:
    <*fmt.wrapError | 0xc00024f2c0>: 
    failed to create NodePool np-fnv7lb: failed waiting for nodepool="np-fnv7lb" for cluster "cluster-candidate-5-0-fnv7lb" in resourcegroup="rg-candidate-5-0-v6nvmt4rxzjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/203ad465-1e1f-4337-9659-f5fe2972a72a",
      "name": "203ad465-1e1f-4337-9659-f5fe2972a72a",
      "status": "Failed",
      "startTime": "2026-09-28T21:17:06.138034027Z",
      "endTime": "2026-09-28T21:36:06.809868093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-v6nvmt4rxzjw-candidate-fnv7lb--managed/providers/Microsoft.Compute/virtualMachines/w5k5h9q4v3j3g4r-np-fnv7lb-b5wnn-tkxbb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00116fc20>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kj25nlz5xd2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "name": "f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "status": "Failed",
      "startTime": "2026-09-27T07:29:57.086711017Z",
      "endTime": "2026-09-27T07:48:57.003548378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00116fc20>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kj25nlz5xd2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "name": "f3fb7911-3b45-4355-a4ce-119b44f62f9a",
      "status": "Failed",
      "startTime": "2026-09-27T07:29:57.086711017Z",
      "endTime": "2026-09-27T07:48:57.003548378Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-kj25nlz5xd2c--managed/providers/Microsoft.Compute/virtualMachines/j7g2h4t8i8p2b9x-arm64-vm-np-1-h8dgh-s6v4c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2104667201544916992
n/a2104104984143466496
cluster CA data must contain exactly one PEM block
full failure pattern:
cluster CA data must contain exactly one PEM block
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e26.67%2 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 2 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6010>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6010>: ...
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2102710804108283904
n/a2102650382160236544
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), st...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.33%1 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 2 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000914940>: 
    failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "status": "Failed",
      "startTime": "2026-09-22T21:22:56.593640254Z",
      "endTime": "2026-09-22T21:51:58.372098632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000914940>: 
    failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "status": "Failed",
      "startTime": "2026-09-22T21:22:56.593640254Z",
      "endTime": "2026-09-22T21:51:58.372098632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc00102a600>: 
    failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "status": "Failed",
      "startTime": "2026-09-22T21:07:20.636049246Z",
      "endTime": "2026-09-22T21:36:27.958431734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc00102a600>: 
    failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "status": "Failed",
      "startTime": "2026-09-22T21:07:20.636049246Z",
      "endTime": "2026-09-22T21:36:27.958431734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2102492733552726016
ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: Conflict; detail message Cannot update resource while resource is "provisioning"; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.33%1 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx"
Unexpected error:
    <*fmt.wrapError | 0xc000c10900>: 
    failed to create NodePool np-pt5qmx: failed starting nodepool creation "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx" in resourcegroup="rg-candidate-5-0-9clwz8rf6599": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx"
Unexpected error:
    <*fmt.wrapError | 0xc000c10900>: 
    failed to create NodePool np-pt5qmx: failed starting nodepool creation "np-pt5qmx" for cluster "cluster-candidate-5-0-pt5qmx" in resourcegroup="rg-candidate-5-0-9clwz8rf6599": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"provisioning\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-9clwz8rf6599/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-pt5qmx/nodePools/np-pt5qmx"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103361291237199872
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.33%1 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001d427a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2f88crxn77t5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b648d347-dd3d-4511-8fd6-84bc235918eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b648d347-dd3d-4511-8fd6-84bc235918eb",
      "name": "b648d347-dd3d-4511-8fd6-84bc235918eb",
      "status": "Failed",
      "startTime": "2026-09-26T22:43:43.636760265Z",
      "endTime": "2026-09-26T23:02:47.067520117Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine l4p0g2x0p8h2h6n-autoscale-noaz-ncv9k-skqk8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine l4p0g2x0p8h2h6n-autoscale-noaz-ncv9k-skqk8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001d427a0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2f88crxn77t5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b648d347-dd3d-4511-8fd6-84bc235918eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b648d347-dd3d-4511-8fd6-84bc235918eb",
      "name": "b648d347-dd3d-4511-8fd6-84bc235918eb",
      "status": "Failed",
      "startTime": "2026-09-26T22:43:43.636760265Z",
      "endTime": "2026-09-26T23:02:47.067520117Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine l4p0g2x0p8h2h6n-autoscale-noaz-ncv9k-skqk8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine l4p0g2x0p8h2h6n-autoscale-noaz-ncv9k-skqk8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103956778688450560
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.33%1 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000f18e80>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-p9kv2wz66ng2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1",
      "name": "82d3308e-222c-40d8-a1be-8c057c26bab1",
      "status": "Failed",
      "startTime": "2026-09-25T15:45:43.486812846Z",
      "endTime": "2026-09-25T16:14:49.560161271Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000f18e80>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-p9kv2wz66ng2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82d3308e-222c-40d8-a1be-8c057c26bab1",
      "name": "82d3308e-222c-40d8-a1be-8c057c26bab1",
      "status": "Failed",
      "startTime": "2026-09-25T15:45:43.486812846Z",
      "endTime": "2026-09-25T16:14:49.560161271Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103500347635929088
VerifyBasicAccess failed: failed to list services: Get "<url>
full failure pattern:
VerifyBasicAccess failed: failed to list services: Get "<url>
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.33%1 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:155]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc00158c4c8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/api/v1/namespaces/default/services": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:155]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc00158c4c8>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/api/v1/namespaces/default/services": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.ingress-cert-le.6r8h.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.166.38:32939->172.30.0.10:53: i/o timeout
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103136634483511296
failed to locate cilium chart: Get "<url>": dial tcp <ip>: connect: network is unreachable
full failure pattern:
failed to locate cilium chart: Get "<url>": dial tcp <ip>: connect: network is unreachable
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.33%1 of 30 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 1 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm
Unexpected error:
    <*fmt.wrapError | 0xc0008581e0>: 
    failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm
Unexpected error:
    <*fmt.wrapError | 0xc0008581e0>: 
    failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2102492733552726016

Environment: PROD

Failure patterns: 22 · Failures matched: 176 · Total runs: 28 · Runs affected: 21

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e1035.71%10 of 28 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 21: 1 · Sep 22: 1 · Sep 23: 2 · Sep 24: 3 · Sep 25: 2 · Sep 26: 3 · Sep 27: 1 · Sep 28: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000b13b60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-6k9kth8pvmf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "name": "eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "status": "Failed",
      "startTime": "2026-09-27T02:13:34.441508798Z",
      "endTime": "2026-09-27T02:32:39.228262555Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0xc000b13b60>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-6k9kth8pvmf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "name": "eb2dcee5-7fa0-489a-bdf1-3568cea3218a",
      "status": "Failed",
      "startTime": "2026-09-27T02:13:34.441508798Z",
      "endTime": "2026-09-27T02:32:39.228262555Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00138eac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-vn4grw565jrj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f58a11a-50d1-416b-9f82-356e8273b038
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f58a11a-50d1-416b-9f82-356e8273b038",
      "name": "7f58a11a-50d1-416b-9f82-356e8273b038",
      "status": "Failed",
      "startTime": "2026-09-26T23:14:10.630574624Z",
      "endTime": "2026-09-26T23:33:16.736724619Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00138eac0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-vn4grw565jrj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f58a11a-50d1-416b-9f82-356e8273b038
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7f58a11a-50d1-416b-9f82-356e8273b038",
      "name": "7f58a11a-50d1-416b-9f82-356e8273b038",
      "status": "Failed",
      "startTime": "2026-09-26T23:14:10.630574624Z",
      "endTime": "2026-09-26T23:33:16.736724619Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0010653a0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-5jg8xwpwj9d7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a82865e3-4228-4f7d-bc65-f5af89262225
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a82865e3-4228-4f7d-bc65-f5af89262225",
      "name": "a82865e3-4228-4f7d-bc65-f5af89262225",
      "status": "Failed",
      "startTime": "2026-09-26T23:13:30.607288907Z",
      "endTime": "2026-09-26T23:32:36.744317191Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0010653a0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-5jg8xwpwj9d7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a82865e3-4228-4f7d-bc65-f5af89262225
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a82865e3-4228-4f7d-bc65-f5af89262225",
      "name": "a82865e3-4228-4f7d-bc65-f5af89262225",
      "status": "Failed",
      "startTime": "2026-09-26T23:13:30.607288907Z",
      "endTime": "2026-09-26T23:32:36.744317191Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (12)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (10)
Date (UTC)Associated PRProw job
n/a2104028667012190208
n/a2103983023094501376
n/a2103665929874313216
n/a2103303533200674816
n/a2103267802377883648
n/a2103258234201575424
n/a2103224303930576896
n/a2102895842133282816
n/a2102578870526939136
n/a2102533571620114432
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 8 days
e2e725.00%7 of 28 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 8 daysShow trend detailsSep 21: 4 · Sep 22: 4 · Sep 23: 4 · Sep 24: 4 · Sep 25: 4 · Sep 26: 2 · Sep 27: 4 · Sep 28: 4STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001048440>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-k6t82b" resourcegroup="rg-np-dg-minor-k6t82b-4f2hjngd28k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-k6t82b-4f2hjngd28k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-k6t82b/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001048440>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-k6t82b" resourcegroup="rg-np-dg-minor-k6t82b-4f2hjngd28k4": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-k6t82b-4f2hjngd28k4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-k6t82b/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000ade640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-t7jl46" in resourcegroup="rg-np-version-upgrade-t7jl46-65khsnvbc4zl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t7jl46-65khsnvbc4zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t7jl46/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000ade640>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-t7jl46" in resourcegroup="rg-np-version-upgrade-t7jl46-65khsnvbc4zl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t7jl46-65khsnvbc4zl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t7jl46/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9"
Unexpected error:
    <*fmt.wrapError | 0xc000d24340>: 
    failed to create NodePool np-5cw4w9: failed starting nodepool creation "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9" in resourcegroup="rg-nightly-4-20-b7gtl2svm55n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-b7gtl2svm55n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-5cw4w9/nodePools/np-5cw4w9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9"
Unexpected error:
    <*fmt.wrapError | 0xc000d24340>: 
    failed to create NodePool np-5cw4w9: failed starting nodepool creation "np-5cw4w9" for cluster "cluster-nightly-4-20-5cw4w9" in resourcegroup="rg-nightly-4-20-b7gtl2svm55n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-b7gtl2svm55n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-5cw4w9/nodePools/np-5cw4w9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.207
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest7
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest6
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2104707837916614656
n/a2104345445676355584
n/a2103983023094501376
n/a2103620630963294208
n/a2103258234201575424
n/a2102895842133282816
n/a2102533571620114432
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 8 days
e2e725.00%7 of 28 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 8 daysShow trend detailsSep 21: 1 · Sep 22: 1 · Sep 23: 1 · Sep 24: 1 · Sep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f35488>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f35488>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014b70f8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014b70f8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc0015174b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.000s.
Expected success, but got an error:
    <*errors.joinError | 0xc0015174b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor7
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2104707837916614656
n/a2104345445676355584
n/a2103983023094501376
n/a2103620630963294208
n/a2103258234201575424
n/a2102895842133282816
n/a2102533571620114432
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e621.43%6 of 28 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 21: 5 · Sep 22: 5 · Sep 23: 4 · Sep 24: 5 · Sep 25: 5 · Sep 26: 5 · Sep 27: 0 · Sep 28: 5STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a51a10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mnwb4zfspsdb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc000a51a10>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mnwb4zfspsdb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00179cc60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-v4rx8l2tw9wq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00179cc60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-v4rx8l2tw9wq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000aa43c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qgqdfprzx89s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc000aa43c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-qgqdfprzx89s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors5
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2104707837916614656
n/a2103983023094501376
n/a2103620630963294208
n/a2103258234201575424
n/a2102895842133282816
n/a2102533571620114432
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e517.86%5 of 28 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4DEV
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007275a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdcdkl9rvn5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612",
      "name": "3a70e081-8150-479f-812b-31eb02522612",
      "status": "Failed",
      "startTime": "2026-09-29T01:01:02.974562787Z",
      "endTime": "2026-09-29T01:20:06.931508754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0007275a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdcdkl9rvn5l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3a70e081-8150-479f-812b-31eb02522612",
      "name": "3a70e081-8150-479f-812b-31eb02522612",
      "status": "Failed",
      "startTime": "2026-09-29T01:01:02.974562787Z",
      "endTime": "2026-09-29T01:20:06.931508754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3s5a5u1s5o4h9w-np-deletable-9nx5r-bcsqf: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc001055c80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tf5tm29lbrgl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "name": "51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "status": "Failed",
      "startTime": "2026-09-28T23:10:33.615095623Z",
      "endTime": "2026-09-28T23:29:40.114080242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2fhfr-24lbg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2fhfr-24lbg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc001055c80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tf5tm29lbrgl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "name": "51bc36e8-ffb9-42f8-8bc5-c2b8cb5c2386",
      "status": "Failed",
      "startTime": "2026-09-28T23:10:33.615095623Z",
      "endTime": "2026-09-28T23:29:40.114080242Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2fhfr-24lbg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2fhfr-24lbg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tf5tm29lbrgl--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2fhfr-24lbg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0030a64a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kx45mg6gnp52" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b",
      "name": "7aa70862-e406-44df-915d-27100318e68b",
      "status": "Failed",
      "startTime": "2026-09-28T23:56:08.330966898Z",
      "endTime": "2026-09-29T00:15:12.920582798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0030a64a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kx45mg6gnp52" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7aa70862-e406-44df-915d-27100318e68b",
      "name": "7aa70862-e406-44df-915d-27100318e68b",
      "status": "Failed",
      "startTime": "2026-09-28T23:56:08.330966898Z",
      "endTime": "2026-09-29T00:15:12.920582798Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kx45mg6gnp52--managed/providers/Microsoft.Compute/virtualMachines/e4w5q3g8d6k8y5e-np-deletable-rkrhk-wfjcc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to delete a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2104710565229236224
n/a2104707837916614656
n/a2104576811751968768
n/a2103620630963294208
n/a2102941141266599936
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e27.14%2 of 28 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 21: 1 · Sep 22: 1 · Sep 23: 1 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000587ac0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-f9strbqhzqsp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-f9strbqhzqsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000587ac0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-f9strbqhzqsp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-f9strbqhzqsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00103f360>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00103f360>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2102895842133282816
n/a2102533571620114432
storage account log verification failed
full failure pattern:
storage account log verification failed
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e1139.29%11 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 4 · Sep 25: 2 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4z25qr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7lft27tlvnbf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0009fab20>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4z25qr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-7lft27tlvnbf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0009fab20>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxwdj8sw and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-fct27gcqstbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0007a08f0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxwdj8sw and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-fct27gcqstbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0007a08f0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxt4mfpn and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-vg2849dzv4lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0004c67b0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxt4mfpn and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-vg2849dzv4lc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0004c67b0>: 
    storage account log verification failed
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings3
Affected runs (11)
Date (UTC)Associated PRProw job
n/a2104390744759341056
n/a2104345445676355584
n/a2104028667012190208
n/a2103983023094501376
n/a2103665929874313216
n/a2103620630963294208
n/a2103303533200674816
n/a2103267802377883648
n/a2103258234201575424
n/a2103224303930576896
n/a2103171230415720448
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster servi...
full failure pattern:
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e517.86%5 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 21: 0 · Sep 22: 1 · Sep 23: 1 · Sep 24: 1 · Sep 25: 0 · Sep 26: 6 · Sep 27: 28 · Sep 28: 0STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0026392a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pndxf7r8t9rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143",
      "name": "a91e490f-050a-4c07-a58f-af347cb9f143",
      "status": "Failed",
      "startTime": "2026-09-28T00:13:40.174684426Z",
      "endTime": "2026-09-28T00:42:43.628005146Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0026392a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pndxf7r8t9rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a91e490f-050a-4c07-a58f-af347cb9f143",
      "name": "a91e490f-050a-4c07-a58f-af347cb9f143",
      "status": "Failed",
      "startTime": "2026-09-28T00:13:40.174684426Z",
      "endTime": "2026-09-28T00:42:43.628005146Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000866340>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wg62gnrkcq82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "name": "38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:28.308542215Z",
      "endTime": "2026-09-27T23:31:35.969190413Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000866340>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-wg62gnrkcq82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "name": "38c68d19-2950-482f-9bf9-f70f6c7063a4",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:28.308542215Z",
      "endTime": "2026-09-27T23:31:35.969190413Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000992660>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-nwqdmg89c6p9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "name": "7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:36.690582806Z",
      "endTime": "2026-09-27T23:31:45.971437914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000992660>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-nwqdmg89c6p9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "name": "7c1f6eed-e1f4-4411-9a1f-fe2e5ddf1e50",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:36.690582806Z",
      "endTime": "2026-09-27T23:31:45.971437914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (33)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.15
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with Image Registry not present1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to lifecycle and confirm external auth on a cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster using v20260901preview API and verify cluster health1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create an HCP cluster and validate TLS certificates1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should not be able to reuse subnets and NSGs between clusters1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2104345445676355584
n/a2103983023094501376
n/a2103258234201575424
n/a2102895842133282816
n/a2102533571620114432
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e414.29%4 of 28 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4DEV, INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m"
Unexpected error:
    <*fmt.wrapError | 0xc00087e120>: 
    failed to create NodePool np-rcls6m: failed waiting for nodepool="np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m" in resourcegroup="rg-candidate-4-21-92w77dwp5rg8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "name": "a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:31.628142216Z",
      "endTime": "2026-09-29T00:35:38.838060326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m"
Unexpected error:
    <*fmt.wrapError | 0xc00087e120>: 
    failed to create NodePool np-rcls6m: failed waiting for nodepool="np-rcls6m" for cluster "cluster-candidate-4-21-rcls6m" in resourcegroup="rg-candidate-4-21-92w77dwp5rg8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "name": "a337697b-a055-47ea-9ec2-d9638cfc3ab2",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:31.628142216Z",
      "endTime": "2026-09-29T00:35:38.838060326Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-92w77dwp5rg8-candidate-rcls6m--managed/providers/Microsoft.Compute/virtualMachines/q8k7t5p0o1m8l5r-np-rcls6m-wc8xf-q85q8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc00130ef60>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-7g7qln9l2np5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "name": "ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "status": "Failed",
      "startTime": "2026-09-28T23:42:15.584467675Z",
      "endTime": "2026-09-29T00:01:22.906593334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc00130ef60>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-7g7qln9l2np5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "name": "ebbacb44-7cd1-425c-b348-f154c9c9ba46",
      "status": "Failed",
      "startTime": "2026-09-28T23:42:15.584467675Z",
      "endTime": "2026-09-29T00:01:22.906593334Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ea-cluster-ea-np-1-4fbc8-5n6n7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7g7qln9l2np5--managed/providers/Microsoft.Compute/virtualMachines/ea-cluster-ea-np-1-4fbc8-5n6n7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l"
Unexpected error:
    <*fmt.wrapError | 0xc00171b4a0>: 
    failed to create NodePool np-nmsq2l: failed waiting for nodepool="np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l" in resourcegroup="rg-candidate-4-21-lf4lgs4pzp7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975",
      "name": "8303805f-b000-457e-b724-d4df7eac6975",
      "status": "Failed",
      "startTime": "2026-09-28T21:30:59.759776246Z",
      "endTime": "2026-09-28T21:50:00.145843425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l"
Unexpected error:
    <*fmt.wrapError | 0xc00171b4a0>: 
    failed to create NodePool np-nmsq2l: failed waiting for nodepool="np-nmsq2l" for cluster "cluster-candidate-4-21-nmsq2l" in resourcegroup="rg-candidate-4-21-lf4lgs4pzp7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/8303805f-b000-457e-b724-d4df7eac6975",
      "name": "8303805f-b000-457e-b724-d4df7eac6975",
      "status": "Failed",
      "startTime": "2026-09-28T21:30:59.759776246Z",
      "endTime": "2026-09-28T21:50:00.145843425Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-lf4lgs4pzp7j-candidate-nmsq2l--managed/providers/Microsoft.Compute/virtualMachines/y5i6c0j1i3c3z4q-np-nmsq2l-ncgkp-j7cz8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2104713411836252160
n/a2104707837916614656
n/a2104669292057333760
n/a2104667835979534336
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Required value: field cannot be null; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e310.71%3 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 14 · Sep 24: 7 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000ab7ea0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-r6szljgv5m4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-r6szljgv5m4p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000ab7ea0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-r6szljgv5m4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-r6szljgv5m4p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000e90d80>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-pc4xbw2wvdgz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-pc4xbw2wvdgz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000e90d80>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-pc4xbw2wvdgz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-pc4xbw2wvdgz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0015fdec0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-gk9hkpv4f4bq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0015fdec0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-gk9hkpv4f4bq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (14)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster using v20260901preview API and verify cluster health1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2102941141266599936
n/a2102895842133282816
n/a2102578870526939136
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e27.14%2 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 21: 1 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 2 · Sep 28: 0DEV, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-j6wfl7rfmftf/cluster-nightly-4-22-nm6bdd should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b18ea0>: 
    failed to create HCP cluster cluster-nightly-4-22-nm6bdd: failed waiting for cluster="cluster-nightly-4-22-nm6bdd" in resourcegroup="rg-nightly-4-22-j6wfl7rfmftf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "name": "30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.981848903Z",
      "endTime": "2026-09-28T00:26:23.602322446Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-j6wfl7rfmftf/cluster-nightly-4-22-nm6bdd should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b18ea0>: 
    failed to create HCP cluster cluster-nightly-4-22-nm6bdd: failed waiting for cluster="cluster-nightly-4-22-nm6bdd" in resourcegroup="rg-nightly-4-22-j6wfl7rfmftf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "name": "30dbd006-3693-4ff9-83b2-2e342bb7120d",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:15.981848903Z",
      "endTime": "2026-09-28T00:26:23.602322446Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fbjrkv" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000adaa00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-fbjrkv: failed waiting for cluster="cp-ystream-upgrade-4-23-fbjrkv" in resourcegroup="rg-cp-ystream-upgrade-4-23-xs76x2g6dgbl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "name": "2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:39.891778199Z",
      "endTime": "2026-09-28T00:26:43.608164604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fbjrkv" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000adaa00>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-fbjrkv: failed waiting for cluster="cp-ystream-upgrade-4-23-fbjrkv" in resourcegroup="rg-cp-ystream-upgrade-4-23-xs76x2g6dgbl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "name": "2c0d0d6a-5f02-484a-8b51-9407d1b552aa",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:39.891778199Z",
      "endTime": "2026-09-28T00:26:43.608164604Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-7q8vqcfshfst/cluster-nightly-5-1-htcp7l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000acaa60>: 
    failed to create HCP cluster cluster-nightly-5-1-htcp7l: failed waiting for cluster="cluster-nightly-5-1-htcp7l" in resourcegroup="rg-nightly-5-1-7q8vqcfshfst" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "name": "3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "status": "Failed",
      "startTime": "2026-09-25T23:32:03.179098884Z",
      "endTime": "2026-09-26T00:01:10.002965076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-7q8vqcfshfst/cluster-nightly-5-1-htcp7l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000acaa60>: 
    failed to create HCP cluster cluster-nightly-5-1-htcp7l: failed waiting for cluster="cluster-nightly-5-1-htcp7l" in resourcegroup="rg-nightly-5-1-7q8vqcfshfst" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "name": "3dbd0e68-03bb-49be-ad53-7a10f8750a36",
      "status": "Failed",
      "startTime": "2026-09-25T23:32:03.179098884Z",
      "endTime": "2026-09-26T00:01:10.002965076Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2104345445676355584
n/a2103620630963294208
cluster CA data must contain exactly one PEM block
full failure pattern:
cluster CA data must contain exactly one PEM block
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e27.14%2 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 21: 0 · Sep 22: 1 · Sep 23: 1 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0INT
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2102578870526939136
n/a2102533571620114432
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or q...
full failure pattern:
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e27.14%2 of 28 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc000b43f40>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc000b43f40>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001139d00>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001139d00>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2104713411836252160
n/a2104667835979534336
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b"
Unexpected error:
    <*fmt.wrapError | 0xc000b7f080>: 
    failed to create NodePool np-bhxv8b: failed waiting for nodepool="np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b" in resourcegroup="rg-candidate-4-21-mrwxnrjtcxks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "name": "b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:53.990233971Z",
      "endTime": "2026-09-29T00:35:58.987434024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b"
Unexpected error:
    <*fmt.wrapError | 0xc000b7f080>: 
    failed to create NodePool np-bhxv8b: failed waiting for nodepool="np-bhxv8b" for cluster "cluster-candidate-4-21-bhxv8b" in resourcegroup="rg-candidate-4-21-mrwxnrjtcxks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "name": "b5cb7fd8-b127-4539-8778-6e7d5e5f9f3d",
      "status": "Failed",
      "startTime": "2026-09-29T00:16:53.990233971Z",
      "endTime": "2026-09-29T00:35:58.987434024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l9g0x5l7a1n1v5a-np-bhxv8b-5zb4x-km8kk: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000bad000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-64htmh" in resourcegroup="rg-np-noedge-64htmh-c4j9qdqfrq4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "name": "3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "status": "Failed",
      "startTime": "2026-09-29T01:00:25.564607872Z",
      "endTime": "2026-09-29T01:19:27.08336887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: Creating: virtualmachine creating or updating\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000bad000>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-64htmh" in resourcegroup="rg-np-noedge-64htmh-c4j9qdqfrq4j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "name": "3b426bd0-3f4b-4ee7-96a0-7d54b3579d89",
      "status": "Failed",
      "startTime": "2026-09-29T01:00:25.564607872Z",
      "endTime": "2026-09-29T01:19:27.08336887Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-md5vk: Creating: virtualmachine creating or updating\nMachine w1j2z1c7q6g2g6k-npnoedge-4-20-7rq4d-mvq9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb"
Unexpected error:
    <*fmt.wrapError | 0xc0010410e0>: 
    failed to create NodePool np-5m4wlb: failed waiting for nodepool="np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb" in resourcegroup="rg-candidate-4-23-wwwsqs8cghpq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052",
      "name": "91d9873c-e44e-4e44-ba41-ade87f500052",
      "status": "Failed",
      "startTime": "2026-09-29T00:18:19.018330415Z",
      "endTime": "2026-09-29T00:37:28.829514453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: Creating: virtualmachine creating or updating\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb"
Unexpected error:
    <*fmt.wrapError | 0xc0010410e0>: 
    failed to create NodePool np-5m4wlb: failed waiting for nodepool="np-5m4wlb" for cluster "cluster-candidate-4-23-5m4wlb" in resourcegroup="rg-candidate-4-23-wwwsqs8cghpq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/91d9873c-e44e-4e44-ba41-ade87f500052",
      "name": "91d9873c-e44e-4e44-ba41-ade87f500052",
      "status": "Failed",
      "startTime": "2026-09-29T00:18:19.018330415Z",
      "endTime": "2026-09-29T00:37:28.829514453Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-jdc42: Creating: virtualmachine creating or updating\nMachine h9a1g7g5s5w9p9g-np-5m4wlb-4q46f-tdh4q: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104710565229236224
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, INT
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-ml64wq with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0013c02a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-ml64wq: failed waiting for cluster="np-version-upgrade-cluster-ml64wq" in resourcegroup="rg-np-version-upgrade-ml64wq-zkbzk7hvlvck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "name": "2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "status": "Failed",
      "startTime": "2026-09-25T00:48:56.872501334Z",
      "endTime": "2026-09-25T01:18:00.947451943Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-ml64wq with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0013c02a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-ml64wq: failed waiting for cluster="np-version-upgrade-cluster-ml64wq" in resourcegroup="rg-np-version-upgrade-ml64wq-zkbzk7hvlvck" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "name": "2d3c2c93-508a-4ae2-aa7a-7a7d3546ac58",
      "status": "Failed",
      "startTime": "2026-09-25T00:48:56.872501334Z",
      "endTime": "2026-09-25T01:18:00.947451943Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103267802377883648
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104345445676355584
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104345445676355584
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet po...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-q9zwpnrqm4zb/cluster-nightly-5-1-sq64f4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110cda0>: 
    failed to create HCP cluster cluster-nightly-5-1-sq64f4: failed waiting for cluster="cluster-nightly-5-1-sq64f4" in resourcegroup="rg-nightly-5-1-q9zwpnrqm4zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec",
      "name": "25287b3f-b870-4e18-a51f-514432892aec",
      "status": "Failed",
      "startTime": "2026-09-28T23:31:45.913312Z",
      "endTime": "2026-09-29T00:00:52.759469958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-q9zwpnrqm4zb/cluster-nightly-5-1-sq64f4 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110cda0>: 
    failed to create HCP cluster cluster-nightly-5-1-sq64f4: failed waiting for cluster="cluster-nightly-5-1-sq64f4" in resourcegroup="rg-nightly-5-1-q9zwpnrqm4zb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/25287b3f-b870-4e18-a51f-514432892aec",
      "name": "25287b3f-b870-4e18-a51f-514432892aec",
      "status": "Failed",
      "startTime": "2026-09-28T23:31:45.913312Z",
      "endTime": "2026-09-29T00:00:52.759469958Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104707837916614656
ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-<id>'...
full failure pattern:
ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-<id>' since another operation is in progress.; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000ff01b0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-d8dncw5q6xr2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735",
      "name": "4cccec67-451f-41c1-957a-1453bb0ac735",
      "status": "Failed",
      "startTime": "2026-09-25T02:15:20.351974211Z",
      "endTime": "2026-09-25T02:34:26.038545442Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-44b92-qd24g": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-44b92-qd24g" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-d8dncw5q6xr2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-44b92-qd24g/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-qd24g' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:205]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc000ff01b0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-d8dncw5q6xr2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cccec67-451f-41c1-957a-1453bb0ac735",
      "name": "4cccec67-451f-41c1-957a-1453bb0ac735",
      "status": "Failed",
      "startTime": "2026-09-25T02:15:20.351974211Z",
      "endTime": "2026-09-25T02:34:26.038545442Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-44b92-qd24g": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-44b92-qd24g" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-d8dncw5q6xr2--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-44b92-qd24g/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-44b92-qd24g' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103303533200674816
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-tssqn7" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0008d6f80>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-tssqn7: failed waiting for cluster="cp-ystream-upgrade-5-0-tssqn7" in resourcegroup="rg-cp-ystream-upgrade-5-0-s5ql4kptw4ct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "name": "8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:40.978774448Z",
      "endTime": "2026-09-28T00:26:43.612304644Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-tssqn7" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0008d6f80>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-tssqn7: failed waiting for cluster="cp-ystream-upgrade-5-0-tssqn7" in resourcegroup="rg-cp-ystream-upgrade-5-0-s5ql4kptw4ct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "name": "8948f23c-1b13-4b8b-b8b8-d745d0f16492",
      "status": "Failed",
      "startTime": "2026-09-27T23:57:40.978774448Z",
      "endTime": "2026-09-28T00:26:43.612304644Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104345445676355584
ERROR CODE: InvalidRequestContent; detail code Conflict; detail message Cannot request credential while credentials are being revoked; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail code Conflict; detail message Cannot request credential while credentials are being revoked; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 1 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:267]: failed to request new admin credentials after revocation for cluster "admin-cred-lifecycle-7t664q"
Unexpected error:
    <*fmt.wrapErrors | 0xc000847980>: 
    20240610 credential request failed: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rwn8v4w9qpfg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-7t664q/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value",
        "target": "certificateSigningRequest"
      }
    }
    --------------------------------------------------------------------------------
    ; 2026-09-01-preview fallback failed: failed to begin admin credential request for hcpCluster="admin-cred-lifecycle-7t664q" in resourcegroup="admin-credential-lifecycle-test-rwn8v4w9qpfg": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rwn8v4w9qpfg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-7t664q/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while credentials are being revoked",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rwn8v4w9qpfg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-7t664q"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:267]: failed to request new admin credentials after revocation for cluster "admin-cred-lifecycle-7t664q"
Unexpected error:
    <*fmt.wrapErrors | 0xc000847980>: 
    20240610 credential request failed: failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rwn8v4w9qpfg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-7t664q/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value",
        "target": "certificateSigningRequest"
      }
    }
    --------------------------------------------------------------------------------
    ; 2026-09-01-preview fallback failed: failed to begin admin credential request for hcpCluster="admin-cred-lifecycle-7t664q" in resourcegroup="admin-credential-lifecycle-test-rwn8v4w9qpfg": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rwn8v4w9qpfg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-7t664q/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot request credential while credentials are being revoked",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rwn8v4w9qpfg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-7t664q"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103983023094501376
Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again
full failure pattern:
Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.57%1 of 28 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x
Unexpected error:
    <*errors.StatusError | 0xc00103ab40>: 
    Operation cannot be fulfilled on nodes "a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x
Unexpected error:
    <*errors.StatusError | 0xc00103ab40>: 
    Operation cannot be fulfilled on nodes "a8y5x1l0f9x5u8y-cordon-np-8kx7p-97f6x": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103171230415720448

Environment: STG

Failure patterns: 22 · Failures matched: 867 · Total runs: 33 · Runs affected: 31

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHat...
full failure pattern:
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e1236.36%12 of 33 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 108 · Sep 22: 107 · Sep 23: 200 · Sep 24: 178 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-p49qttrgxjfs/cluster-candidate-4-23-h8xv9w should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a0e9e0>: 
    failed to create HCP cluster cluster-candidate-4-23-h8xv9w: failed waiting for cluster="cluster-candidate-4-23-h8xv9w" in resourcegroup="rg-candidate-4-23-p49qttrgxjfs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/775da561-b2d7-4a7a-bbc2-e2713e469817
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-p49qttrgxjfs/cluster-candidate-4-23-h8xv9w should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a0e9e0>: 
    failed to create HCP cluster cluster-candidate-4-23-h8xv9w: failed waiting for cluster="cluster-candidate-4-23-h8xv9w" in resourcegroup="rg-candidate-4-23-p49qttrgxjfs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/775da561-b2d7-4a7a-bbc2-e2713e469817
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-q9n7xf" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0015a82e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-q9n7xf: failed waiting for cluster="cp-ystream-upgrade-5-0-q9n7xf" in resourcegroup="rg-cp-ystream-upgrade-5-0-mbh76rk9sprx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e50bc50d-ed3a-44e5-99bf-5d82436a9665
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-q9n7xf" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0015a82e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-q9n7xf: failed waiting for cluster="cp-ystream-upgrade-5-0-q9n7xf" in resourcegroup="rg-cp-ystream-upgrade-5-0-mbh76rk9sprx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e50bc50d-ed3a-44e5-99bf-5d82436a9665
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000994380>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2v5hnvkbp6nz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ac014577-36dc-4208-bce8-f0e66ebed0a8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000994380>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2v5hnvkbp6nz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ac014577-36dc-4208-bce8-f0e66ebed0a8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (166)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation7
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to list available HCP OpenShift versions and validate response content7
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.205
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.215
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.225
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.235
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.05
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.205
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.205
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.225
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.05
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to update node pool labels and taints4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.204
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled4
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.214
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to list available HCP OpenShift versions and validate response content3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete a nodepool2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to update node pool labels and taints2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.02
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.232
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.02
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.12
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage pull secrets2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with Image Registry not present2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with back-level version 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create node pools with ARM64-based VMs2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to delete an HCP cluster whose managed identities were deleted first2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to lifecycle and confirm external auth on a cluster2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to list available HCP OpenShift versions and validate response content2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster using v20260901preview API and verify cluster health2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create an HCP cluster and validate TLS certificates2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should not be able to perform various invalid operations on cluster resources2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should not be able to reuse subnets and NSGs between clusters2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should respect cluster-wide node limits with nodepool autoscaling2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (12)
Date (UTC)Associated PRProw job
n/a2103121232214888448
n/a2103058579819859968
n/a2103041133276827648
n/a2102988159166976000
n/a2102983195996721152
n/a2102941141295960064
n/a2102904467069341696
n/a2102895842175225856
n/a2102810351396458496
n/a2102578870564687872
n/a2102533571657863168
n/a2102216478680944640
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e515.15%5 of 33 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 3 · Sep 25: 4 · Sep 26: 2 · Sep 27: 1 · Sep 28: 4PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001432880>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lllf2h" resourcegroup="rg-np-dg-minor-lllf2h-lmftzgmmkgfx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lllf2h-lmftzgmmkgfx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lllf2h/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001432880>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-lllf2h" resourcegroup="rg-np-dg-minor-lllf2h-lmftzgmmkgfx": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-lllf2h-lmftzgmmkgfx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-lllf2h/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x"
Unexpected error:
    <*fmt.wrapError | 0xc0008e6340>: 
    failed to create NodePool np-br2c4x: failed starting nodepool creation "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x" in resourcegroup="rg-nightly-4-20-bvth8x7dzstx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-bvth8x7dzstx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-br2c4x/nodePools/np-br2c4x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x"
Unexpected error:
    <*fmt.wrapError | 0xc0008e6340>: 
    failed to create NodePool np-br2c4x: failed starting nodepool creation "np-br2c4x" for cluster "cluster-nightly-4-20-br2c4x" in resourcegroup="rg-nightly-4-20-bvth8x7dzstx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-bvth8x7dzstx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-br2c4x/nodePools/np-br2c4x
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000dbaaa0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-grgfk2" in resourcegroup="rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-grgfk2/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000dbaaa0>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-grgfk2" in resourcegroup="rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-grgfk2-gjc5jfp7zsnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-grgfk2/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.205
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest3
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2104707837958557696
n/a2104345445907042304
n/a2103983023136444416
n/a2103620631026208768
n/a2103258234264489985
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e515.15%5 of 33 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d235a8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000d235a8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-28-094035, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011eadb0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011eadb0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-27-005925, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001257740>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001257740>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.20.0-0.nightly-multi-2026-09-13-024541, targetMinor=4.21.0-0.nightly-multi-2026-08-16-064216) failed: clusterversion status.history has no version in target minor "4.21.0-0.nightly-multi-2026-08-16-064216"
    ...
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2104707837958557696
n/a2104345445907042304
n/a2103983023136444416
n/a2103620631026208768
n/a2103258234264489985
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e412.12%4 of 33 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 5 · Sep 25: 5 · Sep 26: 5 · Sep 27: 0 · Sep 28: 5PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a097a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-swk5ggttxwjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a097a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-swk5ggttxwjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00160fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-hfkhgg4cj7jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00160fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-hfkhgg4cj7jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000b459e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-sf9f9x57dtfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000b459e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-sf9f9x57dtfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request4
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2104707837958557696
n/a2103983023136444416
n/a2103620631026208768
n/a2103258234264489985
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e1030.30%10 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 17 · Sep 27: 1 · Sep 28: 2DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8"
Unexpected error:
    <*fmt.wrapError | 0xc0012916a0>: 
    failed to create NodePool np-dpvlx8: failed waiting for nodepool="np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8" in resourcegroup="rg-nightly-5-0-5jrsqj92ljxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e",
      "name": "eee4d653-3b34-478d-9851-5151f132ff5e",
      "status": "Failed",
      "startTime": "2026-09-29T00:23:56.849718476Z",
      "endTime": "2026-09-29T00:43:01.619582837Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8"
Unexpected error:
    <*fmt.wrapError | 0xc0012916a0>: 
    failed to create NodePool np-dpvlx8: failed waiting for nodepool="np-dpvlx8" for cluster "cluster-nightly-5-0-dpvlx8" in resourcegroup="rg-nightly-5-0-5jrsqj92ljxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eee4d653-3b34-478d-9851-5151f132ff5e",
      "name": "eee4d653-3b34-478d-9851-5151f132ff5e",
      "status": "Failed",
      "startTime": "2026-09-29T00:23:56.849718476Z",
      "endTime": "2026-09-29T00:43:01.619582837Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5jrsqj92ljxf-nightly-dpvlx8--managed/providers/Microsoft.Compute/virtualMachines/j6a4t3c4n6z0s3d-np-dpvlx8-65kgg-vkztb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001873458>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zwldz5957z9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "name": "00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "status": "Failed",
      "startTime": "2026-09-28T23:57:06.99965356Z",
      "endTime": "2026-09-29T00:16:13.233773533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-lwwnr-xksft: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-lwwnr-xksft: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001873458>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zwldz5957z9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "name": "00c46f27-c5be-4ae1-883b-7e21c3ef06fc",
      "status": "Failed",
      "startTime": "2026-09-28T23:57:06.99965356Z",
      "endTime": "2026-09-29T00:16:13.233773533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-lwwnr-xksft: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-lwwnr-xksft: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zwldz5957z9k--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-lwwnr-xksft to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000996660>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-z624hhq7js4c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "name": "c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "status": "Failed",
      "startTime": "2026-09-27T02:15:40.256886606Z",
      "endTime": "2026-09-27T02:34:46.354762492Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000996660>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-z624hhq7js4c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "name": "c94480ab-33b3-4cd2-b8c1-de35683bd4b8",
      "status": "Failed",
      "startTime": "2026-09-27T02:15:40.256886606Z",
      "endTime": "2026-09-27T02:34:46.354762492Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f7a0y8t5i5r6v6p-np-1-8s278-9tdcn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-z624hhq7js4c--managed/providers/Microsoft.Compute/virtualMachines/f7a0y8t5i5r6v6p-np-1-8s278-9tdcn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (19)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (10)
Date (UTC)Associated PRProw job
n/a2104707837958557696
n/a2104028630299447297
n/a2103983023136444416
n/a2103956302358122496
n/a2103923127544188928
n/a2103857488750186496
n/a2103778668198236160
n/a2103725120211128320
n/a2103690040713416704
n/a2103665929907867648
ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.
full failure pattern:
ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created.
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e412.12%4 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 24 · Sep 24: 170 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:73]: failed to create customer resources for v20261001preview cluster
Unexpected error:
    <*fmt.wrapError | 0xc00106f760>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-v20261001-x6gm79" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-v20261001-x6gm79/operationStatuses/08584113797462734569
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-w8qdl2mz4kg9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-7e767rdphst3w\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-12/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:73]: failed to create customer resources for v20261001preview cluster
Unexpected error:
    <*fmt.wrapError | 0xc00106f760>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-v20261001-x6gm79" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-v20261001-x6gm79/operationStatuses/08584113797462734569
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-w8qdl2mz4kg9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-7e767rdphst3w\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-12/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:114]: failed to create customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b13100>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ingress-cert-le-5tmv7b" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ingress-cert-le-5tmv7b/operationStatuses/08584113797516953541
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-qzpjtfdnhg6c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-nhimtfs3fueps\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:114]: failed to create customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b13100>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ingress-cert-le-5tmv7b" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ingress-cert-le-5tmv7b/operationStatuses/08584113797516953541
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-qzpjtfdnhg6c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-nhimtfs3fueps\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:283]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000e7dae0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-4lkc6c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-4lkc6c/operationStatuses/08584113794966440147
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-48zl586s8mmx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:283]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000e7dae0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-4lkc6c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-4lkc6c/operationStatuses/08584113794966440147
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-48zl586s8mmx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (113)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to update node pool labels and taints3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.213
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.232
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.02
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2102988159166976000
n/a2102983195996721152
n/a2102941141295960064
n/a2102904467069341696
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet po...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e39.09%3 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 6 · Sep 27: 41 · Sep 28: 1PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdf28j9mzrjf/cluster-nightly-5-1-ds89w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a3e960>: 
    failed to create HCP cluster cluster-nightly-5-1-ds89w5: failed waiting for cluster="cluster-nightly-5-1-ds89w5" in resourcegroup="rg-nightly-5-1-vdf28j9mzrjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b",
      "name": "4cc98024-5165-40da-b846-ec5ee553964b",
      "status": "Failed",
      "startTime": "2026-09-29T00:13:16.920404751Z",
      "endTime": "2026-09-29T00:42:21.556617536Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdf28j9mzrjf/cluster-nightly-5-1-ds89w5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a3e960>: 
    failed to create HCP cluster cluster-nightly-5-1-ds89w5: failed waiting for cluster="cluster-nightly-5-1-ds89w5" in resourcegroup="rg-nightly-5-1-vdf28j9mzrjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4cc98024-5165-40da-b846-ec5ee553964b",
      "name": "4cc98024-5165-40da-b846-ec5ee553964b",
      "status": "Failed",
      "startTime": "2026-09-29T00:13:16.920404751Z",
      "endTime": "2026-09-29T00:42:21.556617536Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-kq44rx at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a482e0>: 
    failed to create HCP cluster np-skip-minor-kq44rx: failed waiting for cluster="np-skip-minor-kq44rx" in resourcegroup="rg-np-skip-minor-kq44rx-wrs7cdgqmf9c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48",
      "name": "123e618f-af28-40f6-94b5-7da4afca4a48",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:09.384523535Z",
      "endTime": "2026-09-28T00:28:17.690643945Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-kq44rx at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000a482e0>: 
    failed to create HCP cluster np-skip-minor-kq44rx: failed waiting for cluster="np-skip-minor-kq44rx" in resourcegroup="rg-np-skip-minor-kq44rx-wrs7cdgqmf9c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/123e618f-af28-40f6-94b5-7da4afca4a48",
      "name": "123e618f-af28-40f6-94b5-7da4afca4a48",
      "status": "Failed",
      "startTime": "2026-09-27T23:59:09.384523535Z",
      "endTime": "2026-09-28T00:28:17.690643945Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000e02800>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ss8z8mq428kv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "name": "5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.434225001Z",
      "endTime": "2026-09-27T23:31:59.489114255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000e02800>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ss8z8mq428kv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "name": "5b586f4c-dc93-44d1-b18c-28c11d67faf1",
      "status": "Failed",
      "startTime": "2026-09-27T23:02:50.434225001Z",
      "endTime": "2026-09-27T23:31:59.489114255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (41)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.13
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with Image Registry not present1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to delete a nodepool1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to lifecycle and confirm external auth on a cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster using v20260901preview API and verify cluster health1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should create an HCP cluster and validate TLS certificates1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should not be able to reuse subnets and NSGs between clusters1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2104707837958557696
n/a2104345445907042304
n/a2103983023136444416
ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machine...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e39.09%3 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 3 · Sep 26: 1 · Sep 27: 0 · Sep 28: 0DEV, INT
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000b56dc0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-c2cwvz" in resourcegroup="rg-np-noedge-c2cwvz-qj8wgrzd8jpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca",
      "name": "4d8efa06-245b-47cb-9abb-ec98064be1ca",
      "status": "Failed",
      "startTime": "2026-09-26T03:07:12.674137276Z",
      "endTime": "2026-09-26T03:26:19.484267097Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000b56dc0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-c2cwvz" in resourcegroup="rg-np-noedge-c2cwvz-qj8wgrzd8jpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4d8efa06-245b-47cb-9abb-ec98064be1ca",
      "name": "4d8efa06-245b-47cb-9abb-ec98064be1ca",
      "status": "Failed",
      "startTime": "2026-09-26T03:07:12.674137276Z",
      "endTime": "2026-09-26T03:26:19.484267097Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-c2cwvz-qj8wgrzd8jpf-np-ne-c2cwvz--managed/providers/Microsoft.Compute/virtualMachines/s7l7b3o9i1a8y0f-npnoedge-4-20-t278w-8cm8n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000feb180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-jmvth8gmbtdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13",
      "name": "c689250c-de76-4270-9b21-5b67d697be13",
      "status": "Failed",
      "startTime": "2026-09-25T16:52:06.978905427Z",
      "endTime": "2026-09-25T17:11:15.434963979Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000feb180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-jmvth8gmbtdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c689250c-de76-4270-9b21-5b67d697be13",
      "name": "c689250c-de76-4270-9b21-5b67d697be13",
      "status": "Failed",
      "startTime": "2026-09-25T16:52:06.978905427Z",
      "endTime": "2026-09-25T17:11:15.434963979Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-jmvth8gmbtdp--managed/providers/Microsoft.Compute/virtualMachines/y8x7q9b3i3q1x0q-np-1-c8cxv-t7pf8 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000e30fe0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-zw4cd4" in resourcegroup="rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012",
      "name": "995a0744-273b-428e-a7c0-5da145da2012",
      "status": "Failed",
      "startTime": "2026-09-25T17:47:45.849510742Z",
      "endTime": "2026-09-25T18:06:46.935940984Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000e30fe0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-zw4cd4" in resourcegroup="rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/995a0744-273b-428e-a7c0-5da145da2012",
      "name": "995a0744-273b-428e-a7c0-5da145da2012",
      "status": "Failed",
      "startTime": "2026-09-25T17:47:45.849510742Z",
      "endTime": "2026-09-25T18:06:46.935940984Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zw4cd4-2qsd7bvk6zfl-np-upgrade-zw-63ddcd9b/providers/Microsoft.Compute/virtualMachines/p6t3i7s0z2k5u2p-npupgrade-4-20-8tr6b-nknqd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2103665929907867648
n/a2103519896854859776
n/a2103344681369735168
storage account log verification failed
full failure pattern:
storage account log verification failed
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e39.09%3 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 2 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxn248k7 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-hwjzhdrfqdml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000973dd0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoeboxn248k7 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-hwjzhdrfqdml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000973dd0>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4lqltr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-gvcrd9xg4vzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0008ace00>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox4lqltr and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-gvcrd9xg4vzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc0008ace00>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox6npjq8 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-g4qtcfdblkfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000d18d20>: 
    storage account log verification failed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:353]: failed to verify shoebox logs reached storage account shoebox6npjq8 and Event Hub shoebox-eh for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-g4qtcfdblkfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*errors.errorString | 0xc000d18d20>: 
    storage account log verification failed
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2103344681369735168
n/a2103303533246812160
n/a2103258234264489985
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e26.06%2 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 3 · Sep 27: 0 · Sep 28: 0DEV
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014c6168>: 
    resource group "tls-endpoint-cluster-s8s7nfpcrjx5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-s8s7nfpcrjx5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f",
      "name": "add1f9fb-6a8e-4417-8717-b28b0f76087f",
      "status": "Failed",
      "startTime": "2026-09-26T21:34:10.468425235Z",
      "endTime": "2026-09-26T21:58:19.111858781Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2ugn3l1bvjlm39njkp5ap0fh8ek88r still exists (deletion dispatched at 2026-09-26T21:34:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014c6168>: 
    resource group "tls-endpoint-cluster-s8s7nfpcrjx5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-s8s7nfpcrjx5" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/add1f9fb-6a8e-4417-8717-b28b0f76087f",
      "name": "add1f9fb-6a8e-4417-8717-b28b0f76087f",
      "status": "Failed",
      "startTime": "2026-09-26T21:34:10.468425235Z",
      "endTime": "2026-09-26T21:58:19.111858781Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2ugn3l1bvjlm39njkp5ap0fh8ek88r still exists (deletion dispatched at 2026-09-26T21:34:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c140c0>: 
    resource group "tls-endpoint-cluster-tlzh4bzgw4zh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-tlzh4bzgw4zh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8",
      "name": "3547c17b-2baf-4dfa-a1b2-187f3ed372c8",
      "status": "Failed",
      "startTime": "2026-09-26T10:04:17.573277188Z",
      "endTime": "2026-09-26T10:28:26.278499332Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdfqqgc9osqgq67u99r2mbueuta1t still exists (deletion dispatched at 2026-09-26T10:04:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c140c0>: 
    resource group "tls-endpoint-cluster-tlzh4bzgw4zh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-tlzh4bzgw4zh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3547c17b-2baf-4dfa-a1b2-187f3ed372c8",
      "name": "3547c17b-2baf-4dfa-a1b2-187f3ed372c8",
      "status": "Failed",
      "startTime": "2026-09-26T10:04:17.573277188Z",
      "endTime": "2026-09-26T10:28:26.278499332Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdfqqgc9osqgq67u99r2mbueuta1t still exists (deletion dispatched at 2026-09-26T10:04:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a00138>: 
    resource group "pullsecret-test-5vdpdhbfbqdk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5vdpdhbfbqdk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4",
      "name": "82ed94ae-6b5c-42a4-9f90-3280fa2efff4",
      "status": "Failed",
      "startTime": "2026-09-26T10:04:15.110849507Z",
      "endTime": "2026-09-26T10:28:16.325232898Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdg1r6crhcm349aur60m1j1fcbuh2 still exists (deletion dispatched at 2026-09-26T10:04:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001a00138>: 
    resource group "pullsecret-test-5vdpdhbfbqdk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-5vdpdhbfbqdk" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/82ed94ae-6b5c-42a4-9f90-3280fa2efff4",
      "name": "82ed94ae-6b5c-42a4-9f90-3280fa2efff4",
      "status": "Failed",
      "startTime": "2026-09-26T10:04:15.110849507Z",
      "endTime": "2026-09-26T10:28:16.325232898Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t2kdg1r6crhcm349aur60m1j1fcbuh2 still exists (deletion dispatched at 2026-09-26T10:04:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2103956302358122496
n/a2103778668198236160
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'...
full failure pattern:
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 58 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000b7e680>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-d5k7vz9fgmp5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-d5k7vz9fgmp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000b7e680>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-d5k7vz9fgmp5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-d5k7vz9fgmp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-5zxmn6" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e7afc0>: 
    failed to create HCP cluster cluster-zstream-4-20-5zxmn6: failed starting cluster creation "cluster-zstream-4-20-5zxmn6" in resourcegroup="rg-zstream-upgrade-4-20-87qdz9hglgx4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-87qdz9hglgx4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-5zxmn6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-5zxmn6" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e7afc0>: 
    failed to create HCP cluster cluster-zstream-4-20-5zxmn6: failed starting cluster creation "cluster-zstream-4-20-5zxmn6" in resourcegroup="rg-zstream-upgrade-4-20-87qdz9hglgx4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-87qdz9hglgx4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-5zxmn6
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-25fcqw
Unexpected error:
    <*fmt.wrapError | 0xc000f12e80>: 
    failed to create HCP cluster np-noedge-25fcqw: failed starting cluster creation "np-noedge-25fcqw" in resourcegroup="rg-np-noedge-25fcqw-mkk44mbtzfcl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-25fcqw-mkk44mbtzfcl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-25fcqw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-25fcqw
Unexpected error:
    <*fmt.wrapError | 0xc000f12e80>: 
    failed to create HCP cluster np-noedge-25fcqw: failed starting cluster creation "np-noedge-25fcqw" in resourcegroup="rg-np-noedge-25fcqw-mkk44mbtzfcl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-25fcqw-mkk44mbtzfcl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-25fcqw
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: LocationNotAvailableForResourceType
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LocationNotAvailableForResourceType",
        "message": "The provided location 'westus3' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (58)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete an HCP cluster whose managed identities were deleted first1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103214845615673344
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000d07ae0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-xxwmzws6kxxc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "name": "85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "status": "Failed",
      "startTime": "2026-09-24T23:13:56.752561651Z",
      "endTime": "2026-09-24T23:32:59.664565124Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0xc000d07ae0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-xxwmzws6kxxc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "name": "85216ecc-6f15-4ee4-9411-ab10183b2cec",
      "status": "Failed",
      "startTime": "2026-09-24T23:13:56.752561651Z",
      "endTime": "2026-09-24T23:32:59.664565124Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and custom node pool osDisk size1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103258234264489985
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104345445907042304
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-wllchtq6tspw/cluster-nightly-5-1-xqrkdk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000614980>: 
    failed to create HCP cluster cluster-nightly-5-1-xqrkdk: failed waiting for cluster="cluster-nightly-5-1-xqrkdk" in resourcegroup="rg-nightly-5-1-wllchtq6tspw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "name": "8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "status": "Failed",
      "startTime": "2026-09-24T23:29:29.214562248Z",
      "endTime": "2026-09-24T23:58:29.739929622Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-wllchtq6tspw/cluster-nightly-5-1-xqrkdk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000614980>: 
    failed to create HCP cluster cluster-nightly-5-1-xqrkdk: failed waiting for cluster="cluster-nightly-5-1-xqrkdk" in resourcegroup="rg-nightly-5-1-wllchtq6tspw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "name": "8d90a29e-3743-4f19-9012-64f6e3adf3af",
      "status": "Failed",
      "startTime": "2026-09-24T23:29:29.214562248Z",
      "endTime": "2026-09-24T23:58:29.739929622Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103258234264489985
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster servi...
full failure pattern:
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-tklhx4zzdx6h/cluster-nightly-5-1-vjz6jm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e262a0>: 
    failed to create HCP cluster cluster-nightly-5-1-vjz6jm: failed waiting for cluster="cluster-nightly-5-1-vjz6jm" in resourcegroup="rg-nightly-5-1-tklhx4zzdx6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e",
      "name": "5f9063e9-f978-4377-b5d9-c9222622088e",
      "status": "Failed",
      "startTime": "2026-09-26T00:24:56.11201607Z",
      "endTime": "2026-09-26T00:54:00.860881838Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-tklhx4zzdx6h/cluster-nightly-5-1-vjz6jm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e262a0>: 
    failed to create HCP cluster cluster-nightly-5-1-vjz6jm: failed waiting for cluster="cluster-nightly-5-1-vjz6jm" in resourcegroup="rg-nightly-5-1-tklhx4zzdx6h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f9063e9-f978-4377-b5d9-c9222622088e",
      "name": "5f9063e9-f978-4377-b5d9-c9222622088e",
      "status": "Failed",
      "startTime": "2026-09-26T00:24:56.11201607Z",
      "endTime": "2026-09-26T00:54:00.860881838Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103620631026208768
ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want >= 2 (autoscaling min); node pool condition AllNodesHealthy is ...
full failure pattern:
ERROR CODE: InternalServerError; detail message [hypershiftNodePool] hypershift NodePool status replicas is 1, want >= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-<id> to exist ; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-<id> to exist; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0015ba9f0>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-2rtwrg4pjxl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714",
      "name": "88abacf9-a5e0-4248-b32c-83d6cd062714",
      "status": "Failed",
      "startTime": "2026-09-25T22:23:02.156910064Z",
      "endTime": "2026-09-25T22:42:06.143243532Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2klmb-6glqn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2klmb-6glqn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0015ba9f0>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-2rtwrg4pjxl2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/88abacf9-a5e0-4248-b32c-83d6cd062714",
      "name": "88abacf9-a5e0-4248-b32c-83d6cd062714",
      "status": "Failed",
      "startTime": "2026-09-25T22:23:02.156910064Z",
      "endTime": "2026-09-25T22:42:06.143243532Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2klmb-6glqn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2klmb-6glqn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-2rtwrg4pjxl2--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2klmb-6glqn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103603313239920640
ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cf4l7z with version 4.21.0-0.nightly-multi-2026-09-24-033657
Unexpected error:
    <*fmt.wrapError | 0xc00142c760>: 
    failed to create HCP cluster np-version-upgrade-cluster-cf4l7z: failed waiting for cluster="np-version-upgrade-cluster-cf4l7z" in resourcegroup="rg-np-version-upgrade-cf4l7z-c8gcs9n4rck7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "name": "58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "status": "Failed",
      "startTime": "2026-09-24T23:42:40.568397466Z",
      "endTime": "2026-09-25T00:11:49.734192333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cf4l7z with version 4.21.0-0.nightly-multi-2026-09-24-033657
Unexpected error:
    <*fmt.wrapError | 0xc00142c760>: 
    failed to create HCP cluster np-version-upgrade-cluster-cf4l7z: failed waiting for cluster="np-version-upgrade-cluster-cf4l7z" in resourcegroup="rg-np-version-upgrade-cf4l7z-c8gcs9n4rck7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "name": "58578399-7a04-4d7e-9fff-65b3ebeb1ba4",
      "status": "Failed",
      "startTime": "2026-09-24T23:42:40.568397466Z",
      "endTime": "2026-09-25T00:11:49.734192333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103258234264489985
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001152660>: 
    resource group "rg-cp-ystream-upgrade-4-22-k576cn8fjplq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-c95x5l" in resourcegroup="rg-cp-ystream-upgrade-4-22-k576cn8fjplq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "name": "5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "status": "Failed",
      "startTime": "2026-09-28T22:54:48.090437049Z",
      "endTime": "2026-09-28T23:18:56.801422937Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t49njjvu97eroh1p7j9jssq535c6590 still exists (deletion dispatched at 2026-09-28T22:54:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001152660>: 
    resource group "rg-cp-ystream-upgrade-4-22-k576cn8fjplq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-c95x5l" in resourcegroup="rg-cp-ystream-upgrade-4-22-k576cn8fjplq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "name": "5533b7af-fbcc-4912-9e0c-66a6d54c8b8b",
      "status": "Failed",
      "startTime": "2026-09-28T22:54:48.090437049Z",
      "endTime": "2026-09-28T23:18:56.801422937Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t49njjvu97eroh1p7j9jssq535c6590 still exists (deletion dispatched at 2026-09-28T22:54:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104687818398765056
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpen...
full failure pattern:
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, uksouth'.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0017589f0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: NoRegisteredProviderFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "NoRegisteredProviderFound",
        "message": "No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0017589f0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: NoRegisteredProviderFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "NoRegisteredProviderFound",
        "message": "No registered resource provider found for location 'westus3' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, uksouth'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to list available HCP OpenShift versions and validate response content1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103214845615673344
Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again
full failure pattern:
Operation cannot be fulfilled on nodes "<node>": the object has been modified; please apply your changes to the latest version and try again
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:214]: failed to cordon node d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6
Unexpected error:
    <*errors.StatusError | 0xc000b17180>: 
    Operation cannot be fulfilled on nodes "d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:214]: failed to cordon node d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6
Unexpected error:
    <*errors.StatusError | 0xc000b17180>: 
    Operation cannot be fulfilled on nodes "d7p3x3q5p9p9x4z-cordon-np-7bdwq-kf6d6": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2103344681369735168
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 0 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 0DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2104345445907042304
failed to create HCP cluster <cluster>: failed starting cluster creation "<cluster>" in resourcegroup="<resource-group>": Put "<url>": stream error: stream ID 37; CANCEL; received...
full failure pattern:
failed to create HCP cluster <cluster>: failed starting cluster creation "<cluster>" in resourcegroup="<resource-group>": Put "<url>": stream error: stream ID 37; CANCEL; received from peer
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e13.03%1 of 33 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 21: 0 · Sep 22: 1 · Sep 23: 0 · Sep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0013940e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0013940e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2102216478680944640