CIHealth
<
Single Day: Oct 2▾
Last 7 Days Weekly: Sep 28 - Oct 4 Sprint: Sep 28 - Oct 11 Single Day: Oct 2
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 32 · Failures matched: 94 · Total runs: 66 · Runs affected: 46

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRkube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e34.55%3 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.006s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-d97ndt" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc001ec6360>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.006s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-d97ndt" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc001ec6360>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.001s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-qjs7pm" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0011ce348>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.001s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-qjs7pm" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0011ce348>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.009s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-dg8wqw" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0015c3050>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:240]: Timed out after 2700.009s.
control plane did not reach 5.1 on cluster "cp-ystream-deny-pods-5-1-dg8wqw" with deny-pod ValidatingAdmissionPolicy
Expected success, but got an error:
    <*errors.joinError | 0xc0015c3050>: 
    kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    clusterversion status.history has no version in target minor "5.1"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #68892106117061750034432
PR #68892106055090056990720
PR #68892105963382757658624
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation "hypershift.openshi...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation "hypershift.openshift.io/cluster-size-override" is "e2e_minimal; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fskm94" on minor version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a32b40>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-fskm94" in resourcegroup="rg-zstream-upgrade-4-20-7d7sxcc5nbp4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "name": "0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:38.87341489Z",
      "endTime": "2026-10-02T23:43:46.612931387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-fskm94" on minor version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a32b40>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-fskm94" in resourcegroup="rg-zstream-upgrade-4-20-7d7sxcc5nbp4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "name": "0a14b50b-454e-42ea-8dd4-4340cfdff6fa",
      "status": "Failed",
      "startTime": "2026-10-02T22:59:38.87341489Z",
      "endTime": "2026-10-02T23:43:46.612931387Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-59v6sn" on minor version 4.23
Unexpected error:
    <*fmt.wrapError | 0xc000be1060>: 
    failed waiting for hcpCluster="cluster-zstream-4-23-59v6sn" in resourcegroup="rg-zstream-upgrade-4-23-dwdrgs9rpkt4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32",
      "name": "f2c90638-d31c-4819-aa00-1f73910cfe32",
      "status": "Failed",
      "startTime": "2026-10-02T23:00:29.412121269Z",
      "endTime": "2026-10-02T23:44:36.589759019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-59v6sn" on minor version 4.23
Unexpected error:
    <*fmt.wrapError | 0xc000be1060>: 
    failed waiting for hcpCluster="cluster-zstream-4-23-59v6sn" in resourcegroup="rg-zstream-upgrade-4-23-dwdrgs9rpkt4" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2c90638-d31c-4819-aa00-1f73910cfe32",
      "name": "f2c90638-d31c-4819-aa00-1f73910cfe32",
      "status": "Failed",
      "startTime": "2026-10-02T23:00:29.412121269Z",
      "endTime": "2026-10-02T23:44:36.589759019Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*fmt.wrapError | 0xc000a6f800>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-xzgkv9b984lz" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "name": "bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:51.980435313Z",
      "endTime": "2026-10-02T23:08:01.293538008Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*fmt.wrapError | 0xc000a6f800>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-xzgkv9b984lz" to finish updating: GET https://rp.j2076800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "name": "bc6e51ed-79e2-431d-8b60-c9c38f639a11",
      "status": "Failed",
      "startTime": "2026-10-02T22:58:51.980435313Z",
      "endTime": "2026-10-02T23:08:01.293538008Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73132106143183632076800
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InvalidRequestContent; detail message Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/<subscription>/resourcegroups/<resource-gro...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Monitor/accounts/services-j7538560; provider Microsoft.Monitor
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0xc0013d1ed0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/services-j7538560"
        }
    }
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:493]: failed to create alert processing rule
Unexpected error:
    <*errors.errorString | 0xc0013d1ed0>: 
    expected status 200, got 400: {
        "error": {
            "code": "InvalidRequestContent",
            "message": "Invalid scope(s) in request: at least one of these scopes is required: /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/hcps-j7538560,/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7538560/providers/Microsoft.Monitor/accounts/services-j7538560"
        }
    }
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to perform alert processing rule CRUD operations1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73102106164805017538560
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRUpdateHCPCluster20251223 timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster20251223 timed out after <minutes> minutes; context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*fmt.wrapErrors | 0xc0013d63c0>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d4f6lxz4s2dz" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-d4f6lxz4s2dz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*fmt.wrapErrors | 0xc0013d63c0>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-d4f6lxz4s2dz" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20251223 for cluster idms-e2e-hcp-cluster in resource group idms-d4f6lxz4s2dz, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892105963382757658624
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRUpdateHCPCluster20261001 timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster20261001 timed out after <minutes> minutes; context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:369]: failed to update containerRegistry via PATCH
Unexpected error:
    <*fmt.wrapErrors | 0xc000fa8690>: 
    failed waiting for hcpCluster="acr-pull-create" in resourcegroup="acr-pull-create-ljhpblch2468" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20261001 for cluster acr-pull-create in resource group acr-pull-create-ljhpblch2468, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:369]: failed to update containerRegistry via PATCH
Unexpected error:
    <*fmt.wrapErrors | 0xc000fa8690>: 
    failed waiting for hcpCluster="acr-pull-create" in resourcegroup="acr-pull-create-ljhpblch2468" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster20261001 for cluster acr-pull-create in resource group acr-pull-create-ljhpblch2468, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892105963382757658624
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRconsole should return a successful response or redirect, got 503
full failure pattern:
console should return a successful response or redirect, got 503
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 2 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:212]: Timed out after 2700.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:210 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:212]: Timed out after 2700.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:210 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68072105987324146880512
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to poll cluster "<cluster>" authorized CIDRs update to completion (timeout '10.000000' minutes)
full failure pattern:
failed to poll cluster "<cluster>" authorized CIDRs update to completion (timeout '10.000000' minutes)
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:416]: failed to poll cluster "cidr-connectivity-test" authorized CIDRs update to completion (timeout '10.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:416]: failed to poll cluster "cidr-connectivity-test" authorized CIDRs update to completion (timeout '10.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892105963382757658624
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to update cluster "<cluster>" authorized CIDRs to include test runner IP
full failure pattern:
failed to update cluster "<cluster>" authorized CIDRs to include test runner IP
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.52%1 of 66 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:282]: failed to update cluster "agg-cluster" authorized CIDRs to include test runner IP
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892105963382757658624
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert1928.79%19 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 26: 0 · Sep 27: 2 · Sep 28: 7 · Sep 29: 11 · Sep 30: 3 · Oct 1: 5 · Oct 2: 19none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:06:25Z
  Ended:       2026-10-03T00:13:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6364800-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:07:24Z
  Ended:       2026-10-03T00:19:25Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7564928-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-03T00:08:27Z
  Ended:       2026-10-03T00:27:27Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7564928-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T21:34:19Z
  Ended:       2026-10-02T21:40:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8116608-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire19
Affected runs (19)
Date (UTC)Associated PRProw job
PR #69282106157602076364800
PR #73312106157368797564928
PR #73272106114529938116608
PR #73262106109176676618240
PR #67752106081367363162112
PR #69762106081336937680896
PR #73232106078475705126912
PR #73092106064405874610176
PR #73262106063651092828160
PR #73212106050489337516032
PR #72322106039082894233600
PR #69762106035454687055872
PR #73232106027308732125184
PR #71432106024077666816000
PR #73092106020738229080064
PR #72922106003628014702592
PR #73092105975083930488832
PR #72932105933227649667072
PR #69762105916967100616704
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert57.58%5 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 26: 1 · Sep 27: 2 · Sep 28: 1 · Sep 29: 7 · Sep 30: 10 · Oct 1: 6 · Oct 2: 5none
Full failure examples (3)
alert fired 9 time(s) (1 unknown, 8 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:22:54Z
  Ended:       2026-10-03T00:28:53Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7564928-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6vba5aukt21s79ka316m1hqn628hho-s4a5z1v1k7m8i6h", pod="openshift-apiserver-64ddcdd5c4-tjv4j", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6vba5aukt21s79ka316m1hqn628hho-s4a5z1v1k7m8i6h/openshift-apiserver-64ddcdd5c4-tjv4j has been in a non-ready state for longer than 5 minutes.
alert fired 40 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T20:04:20Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-rqmvp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-rqmvp has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T20:04:20Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-pclwm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-pclwm has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-ffmdk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-ffmdk has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-lqbwl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-lqbwl has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7xqdb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7xqdb has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-xfn2j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-xfn2j has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-hrb67", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-hrb67 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-9rzfn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-9rzfn has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-8rlch", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-8rlch has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-xz6lm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-xz6lm has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-psr9c", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-psr9c has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-tdg92", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-tdg92 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nnzlj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nnzlj has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-cp4cg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-cp4cg has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-zs2jk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-zs2jk has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-bft9l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-bft9l has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-l6fg8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-l6fg8 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nkfbd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nkfbd has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-zqdt7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-zqdt7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-6hf78", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-6hf78 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-v2424", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-v2424 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7xlbs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7xlbs has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-gr2nd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-gr2nd has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-hhkrx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-hhkrx has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-wnr5j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-wnr5j has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-6kxrp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-6kxrp has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-wxmwp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-wxmwp has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-dgvs2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-dgvs2 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-nt5f5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-nt5f5 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-qldfs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-qldfs has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-vjmrq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-vjmrq has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-mz8lv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-mz8lv has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-7r5rf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-7r5rf has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:23:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-4ljs8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-4ljs8 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-vp8px", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-vp8px has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-t8hh6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-t8hh6 has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-9xv2x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-9xv2x has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-w4rxn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-w4rxn has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a", pod="ignition-server-proxy-789f8b6d5-mfxhq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roevlt7isirqt9vh847qin5e1aq08-z1l9m0v5r8f6l2a/ignition-server-proxy-789f8b6d5-mfxhq has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-02T20:05:22Z
  Ended:       2026-10-02T20:18:21Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3827968-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b", pod="ignition-server-proxy-5fc76ccd5b-vngdx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t6roerb25v628veke8v55ppk077aba6-u0o7a0t6f5u8t3b/ignition-server-proxy-5fc76ccd5b-vngdx has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T17:21:19Z
  Ended:       2026-10-02T17:32:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6990720-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t6pa7oq39hmo04tlm81048rosc9slps-ea-listt98lhd", pod="kube-apiserver-594c7bf48-44xbg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t6pa7oq39hmo04tlm81048rosc9slps-ea-listt98lhd/kube-apiserver-594c7bf48-44xbg has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #73312106157368797564928
PR #73112106091840703827968
PR #68892106055090056990720
PR #67752106035412379111424
PR #68892105963382757658624
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e46.06%4 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 0 · Sep 30: 1 · Oct 1: 0 · Oct 2: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bc2640>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bc2640>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.003s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001e820c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ftx74z7plhzf" to finish creating: GET https://rp.j1487616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "name": "737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "status": "Failed",
      "startTime": "2026-10-02T19:53:28.478249577Z",
      "endTime": "2026-10-02T20:12:35.176603143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001e820c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-ftx74z7plhzf" to finish creating: GET https://rp.j1487616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "name": "737de5ed-e682-479d-bd22-1b9ebdbeab1a",
      "status": "Failed",
      "startTime": "2026-10-02T19:53:28.478249577Z",
      "endTime": "2026-10-02T20:12:35.176603143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cmpgt-5gqm9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-5gqm9 to exist\nMachine cilium-cluster-cilium-np-cmpgt-cg7p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ftx74z7plhzf--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cmpgt-cg7p2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bdc160>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:296]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
    <*fmt.wrapError | 0xc000bdc160>: 
    VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.001s: failed to list pods in namespace pullsecret-image-test: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets3
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #72782106141275559301120
PR #73212106096143061487616
PR #72322106039082894233600
PR #69762105989639176196096
cluster entered terminal Failed state after role assignment deployment
full failure pattern:
cluster entered terminal Failed state after role assignment deployment
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e34.55%3 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 26: 4 · Sep 27: 4 · Sep 28: 7 · Sep 29: 1 · Sep 30: 0 · Oct 1: 0 · Oct 2: 3none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #73262106171709915664384
PR #73112106091840703827968
PR #73052106082635410313216
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert34.55%3 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 26: 2 · Sep 27: 1 · Sep 28: 5 · Sep 29: 6 · Sep 30: 7 · Oct 1: 0 · Oct 2: 3none
Full failure examples (3)
alert fired 65 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-22dl9", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-22dl9 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dfssq", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dfssq has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-lc7f8", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-lc7f8 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-lrxrn", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-lrxrn has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ws459", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ws459 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-nm4b8", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-nm4b8 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-r88sm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-r88sm has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-2f87w", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-2f87w has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-hlcf4", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-hlcf4 has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6mnrg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6mnrg has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-t95zp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-t95zp has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jkbjb", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jkbjb has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zcpgg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zcpgg has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-qxvct", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-qxvct has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-mfbkm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-mfbkm has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-mmpwv", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-mmpwv has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-m4v82", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-m4v82 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-fzf98", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-fzf98 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6f2b7", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6f2b7 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-z9cp6", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-z9cp6 has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-64mk2", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-64mk2 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-glxpz", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-glxpz has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-scxrj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-scxrj has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5876w", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5876w has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-h5skh", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-h5skh has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-77rvv", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-77rvv has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-wxzgc", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-wxzgc has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gxvrp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gxvrp has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-f497d", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-f497d has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zpfft", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zpfft has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-x99lx", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-x99lx has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ksr8r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ksr8r has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-p2k9c", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-p2k9c has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-hp5fb", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-hp5fb has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vjh72", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vjh72 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gfrgk", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gfrgk has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-nrh6c", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-nrh6c has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dcz7s", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dcz7s has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-56n2t", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-56n2t has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-8pwww", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-8pwww has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-sr95r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-sr95r has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-sl2gh", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-sl2gh has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-dlbhp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-dlbhp has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-pbjdw", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-pbjdw has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-rfjgp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-rfjgp has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-xwfwj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-xwfwj has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-gl2xg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-gl2xg has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-96lg5", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-96lg5 has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vbcxj", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vbcxj has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5sw59", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5sw59 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-l49jm", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-l49jm has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jx5v4", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jx5v4 has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-s256n", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-s256n has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-vnr48", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-vnr48 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:46Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-zrk5n", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-zrk5n has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-cc7rs", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-cc7rs has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-6j84r", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-6j84r has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-blmqs", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-blmqs has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-4lnm2", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-4lnm2 has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-ppjmp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-ppjmp has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-t9nwg", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-t9nwg has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-68bhc", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-68bhc has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-5fhnt", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-5fhnt has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-jjrn5", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-jjrn5 has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Fired (not resolved)
  Started:     2026-10-02T17:38:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2828160-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-rzbk6", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-rzbk6 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T15:56:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9111424-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5685964bfc-k7gkp", region="canadacentral", severity="warning"
  Description: Pod monitoring/kube-events-5685964bfc-k7gkp has been in a non-ready state for longer than 5 minutes.
alert fired 5 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-wzq9v", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-wzq9v has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-qj4ln", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-qj4ln has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-w44fw", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-w44fw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="arohcp-monitor-kube-state-metrics-79b88f5699-66rf9", region="canadacentral", severity="warning"
  Description: Pod prometheus/arohcp-monitor-kube-state-metrics-79b88f5699-66rf9 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-02T04:11:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3304448-svc", component="kubernetes-infrastructure", namespace="prometheus", pod="prometheus-operator-5d45794ff-4p6tc", region="canadacentral", severity="warning"
  Description: Pod prometheus/prometheus-operator-5d45794ff-4p6tc has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #73262106063651092828160
PR #67752106035412379111424
PR #71052105860601363304448
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert11.52%1 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 26: 1 · Sep 27: 1 · Sep 28: 1 · Sep 29: 5 · Sep 30: 2 · Oct 1: 5 · Oct 2: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:19:46Z
  Ended:       2026-10-02T12:37:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-02T12:19:46Z
  Ended:       2026-10-02T12:37:46Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73032105975953711697920
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert11.52%1 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 26: 1 · Sep 27: 0 · Sep 28: 5 · Sep 29: 3 · Sep 30: 4 · Oct 1: 5 · Oct 2: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:24:46Z
  Ended:       2026-10-02T12:39:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jkdxm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73032105975953711697920
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert11.52%1 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 26: 1 · Sep 27: 0 · Sep 28: 1 · Sep 29: 2 · Sep 30: 3 · Oct 1: 5 · Oct 2: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-02T12:11:46Z
  Ended:       2026-10-02T12:41:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3ec768d5-6cf8-4b9c-a960-342894bf7da7"
  Description: Pod ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-02T12:11:46Z
  Ended:       2026-10-02T12:41:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1697920-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1697920/providers/microsoft.monitor/accounts/hcps-j1697920", namespace="ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="3ec768d5-6cf8-4b9c-a960-342894bf7da7"
  Description: Pod ocm-arohcpci01-2t6kocm5nlh2bsj6qar6cn2cp8c1dutp-j3g1z4j4b8m5s0b/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73032105975953711697920
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e11.52%1 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 26: 0 · Sep 27: 2 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002838820>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc002838820>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73112106091840703827968
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e11.52%1 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 26: 0 · Sep 27: 1 · Sep 28: 0 · Sep 29: 1 · Sep 30: 3 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x152c481045e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s55djgjncgbr" to finish creating: GET https://rp.j5664384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x152c481045e0>: 
    failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s55djgjncgbr" to finish creating: GET https://rp.j5664384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bd7ec69c-0275-4790-bd33-ff2503c1f27a"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73262106171709915664384
private ingress should not be reachable from outside the VNet, but connection succeeded
full failure pattern:
private ingress should not be reachable from outside the VNet, but connection succeeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e11.52%1 of 66 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 26: 1 · Sep 27: 0 · Sep 28: 0 · Sep 29: 2 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72932105830212750020608
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e1928.79%19 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 26: 2 · Sep 27: 7 · Sep 28: 19 · Sep 29: 10 · Sep 30: 8 · Oct 1: 21 · Oct 2: 21none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz"
Unexpected error:
    <*fmt.wrapError | 0xc001182600>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz" in resourcegroup="rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt" to finish creating: GET https://rp.j6364800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "name": "6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "status": "Failed",
      "startTime": "2026-10-03T00:01:24.493949107Z",
      "endTime": "2026-10-03T00:20:29.737570785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz"
Unexpected error:
    <*fmt.wrapError | 0xc001182600>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-0-sd9hfz" in resourcegroup="rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt" to finish creating: GET https://rp.j6364800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "name": "6ca047c0-02df-4aac-a493-0a13f044d7dc",
      "status": "Failed",
      "startTime": "2026-10-03T00:01:24.493949107Z",
      "endTime": "2026-10-03T00:20:29.737570785Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-dgc2p to exist\nMachine m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-0-cmdz5pmslwlt-cp-ystream-den-95628066/providers/Microsoft.Compute/virtualMachines/m5o4v4c8c8p6v5f-np-deny-pods-wc6td-tljpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ad2a60>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9h4twv" in resourcegroup="rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx" to finish creating: GET https://rp.j9301120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072",
      "name": "798eb1c5-8054-497c-b466-6bdb523be072",
      "status": "Failed",
      "startTime": "2026-10-02T23:16:07.311984652Z",
      "endTime": "2026-10-02T23:35:13.399526187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.40
Unexpected error:
    <*fmt.wrapError | 0xc000ad2a60>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-9h4twv" in resourcegroup="rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx" to finish creating: GET https://rp.j9301120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/798eb1c5-8054-497c-b466-6bdb523be072",
      "name": "798eb1c5-8054-497c-b466-6bdb523be072",
      "status": "Failed",
      "startTime": "2026-10-02T23:16:07.311984652Z",
      "endTime": "2026-10-02T23:35:13.399526187Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9h4twv-s2qpdtw4l5dx-np-upgrade-9h-7e5530d9/providers/Microsoft.Compute/virtualMachines/p5q9c9u0b2w8w7g-npupgrade-4-20-2vngn-hhgjs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xc0008d1b20>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-b86qkn" in resourcegroup="rg-np-version-upgrade-b86qkn-wdjflpkjvp8p" to finish creating: GET https://rp.j0034432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24",
      "name": "ba8abfad-4308-4775-a81e-7fccc827da24",
      "status": "Failed",
      "startTime": "2026-10-02T21:59:34.896590709Z",
      "endTime": "2026-10-02T22:18:38.199181809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.35
Unexpected error:
    <*fmt.wrapError | 0xc0008d1b20>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-b86qkn" in resourcegroup="rg-np-version-upgrade-b86qkn-wdjflpkjvp8p" to finish creating: GET https://rp.j0034432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba8abfad-4308-4775-a81e-7fccc827da24",
      "name": "ba8abfad-4308-4775-a81e-7fccc827da24",
      "status": "Failed",
      "startTime": "2026-10-02T21:59:34.896590709Z",
      "endTime": "2026-10-02T22:18:38.199181809Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b86qkn-wdjflpkjvp8p-np-upgrade-b8-539690a9/providers/Microsoft.Compute/virtualMachines/j2s1d7t0q5w9c1v-npupgrade-4-21-cfbt7-68vsc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (17)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor1
Affected runs (19)
Date (UTC)Associated PRProw job
PR #69282106157602076364800
PR #72782106141275559301120
PR #68892106117061750034432
PR #73272106114529938116608
PR #73092106109131696902144
PR #73052106082635410313216
PR #69762106081336937680896
PR #73262106063651092828160
PR #72322106039082894233600
PR #73142106035428787228672
PR #67752106035412379111424
PR #73232106027308732125184
PR #67752105989665596116992
PR #73092105975083930488832
PR #68892105963382757658624
PR #71052105899086866550784
PR #73142105860632891887616
PR #67752105860557423775744
PR #71052105811761943810048
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e34.55%3 of 66 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 41 · Sep 30: 1 · Oct 1: 0 · Oct 2: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001047368>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc001047368>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011448e8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc0011448e8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000dc15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000dc15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #68892106117061750034432
PR #68892106055090056990720
PR #68892105963382757658624
alert [svc] KustoLogsDataStale fired
full failure pattern:
alert [svc] KustoLogsDataStale fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
alert23.03%2 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 2 · Sep 30: 2 · Oct 1: 3 · Oct 2: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T22:50:00Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0034432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0034432/providers/microsoft.monitor/accounts/services-j0034432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-fdbccb774-c7nq9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0034432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T22:50:00Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0034432-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.34:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0034432/providers/microsoft.monitor/accounts/services-j0034432", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-fdbccb774-c7nq9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j0034432-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-02T19:11:26Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4610176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4610176/providers/microsoft.monitor/accounts/services-j4610176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-69f9b555fd-7q6cb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4610176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-02T19:11:26Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4610176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4610176/providers/microsoft.monitor/accounts/services-j4610176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-69f9b555fd-7q6cb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4610176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #68892106117061750034432
PR #73092106064405874610176
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e23.03%2 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00378d080>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-46hf4d98wdxd" to finish creating: GET https://rp.j6618240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "name": "19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "status": "Failed",
      "startTime": "2026-10-02T21:11:48.464360183Z",
      "endTime": "2026-10-02T21:30:51.922878069Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00378d080>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-46hf4d98wdxd" to finish creating: GET https://rp.j6618240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "name": "19cdee0b-c4b9-46e1-8ec8-1661a96d6bea",
      "status": "Failed",
      "startTime": "2026-10-02T21:11:48.464360183Z",
      "endTime": "2026-10-02T21:30:51.922878069Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-46hf4d98wdxd--managed/providers/Microsoft.Compute/virtualMachines/s0m9z3c4v3c7d8p-np-delete-main-pqbrm-hxrzm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00063e600>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-snc8bz4l2lpm" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999",
      "name": "377e91a7-9fb9-4364-a4c3-94d301101999",
      "status": "Failed",
      "startTime": "2026-10-02T11:41:21.075539741Z",
      "endTime": "2026-10-02T12:00:23.993693871Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00063e600>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-snc8bz4l2lpm" to finish creating: GET https://rp.j7658624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/377e91a7-9fb9-4364-a4c3-94d301101999",
      "name": "377e91a7-9fb9-4364-a4c3-94d301101999",
      "status": "Failed",
      "startTime": "2026-10-02T11:41:21.075539741Z",
      "endTime": "2026-10-02T12:00:23.993693871Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-snc8bz4l2lpm--managed/providers/Microsoft.Compute/virtualMachines/k8j2o5c1s9g3f7l-np-delete-main-gpk8z-64bzj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73262106109176676618240
PR #68892105963382757658624
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e23.03%2 of 66 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1 · Oct 1: 0 · Oct 2: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0028843f0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kqrgzxwnxwgd" to finish creating: GET https://rp.j8271104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "name": "7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "status": "Failed",
      "startTime": "2026-10-02T12:59:29.315762309Z",
      "endTime": "2026-10-02T13:18:36.006994146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-597b9-prdsc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-597b9-prdsc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc0028843f0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kqrgzxwnxwgd" to finish creating: GET https://rp.j8271104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "name": "7b30f413-12b5-4064-9da7-46a76bf0cf7b",
      "status": "Failed",
      "startTime": "2026-10-02T12:59:29.315762309Z",
      "endTime": "2026-10-02T13:18:36.006994146Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-597b9-prdsc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-597b9-prdsc: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000b84f00>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2vzgl45q9ll4" to finish creating: GET https://rp.j0616704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "name": "992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "status": "Failed",
      "startTime": "2026-10-02T08:27:53.917230714Z",
      "endTime": "2026-10-02T08:46:59.056787427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000b84f00>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2vzgl45q9ll4" to finish creating: GET https://rp.j0616704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "name": "992f2fbe-f68f-43d2-ae3a-903d3d70bb71",
      "status": "Failed",
      "startTime": "2026-10-02T08:27:53.917230714Z",
      "endTime": "2026-10-02T08:46:59.056787427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l4k0b7t8n9c5i1g-np-1-4f2zv-4fztr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73142105989632788271104
PR #69762105916967100616704
failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1 context deadline exceeded
full failure pattern:
failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1 context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision23.03%2 of 66 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 2none
Full failure examples (2)
time=2026-10-02T04:18:24.964Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-02T04:18:25.604Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:3a6b5dce-c503-483f-868b-3656736645ed cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j6706176.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j6706176-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j6706176-1 secret:hcp-underlay-j6706176-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-02T04:18:25.604Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:18:26.032Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-02T04:18:26.347Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-02T04:18:26.347Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:18:26.470Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:23:32.794Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:23:32.795Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:04:12.931Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-10-02T04:04:13.290Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 values="map[azure:map[clientId:ad930020-a0d3-4a84-b682-06665523a404 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-ci01-maestro-j3304448.canadacentral-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j3304448-mgmt-1 credsKeyVault:map[name:ah-ci01-mg-j3304448-1 secret:hcp-underlay-j3304448-mgmt-1] deployment:map[zoneCount:3] glog_v:2 image:map[digest:sha256:9af415e086251e94f471e03844ce768bed6493653b6cabdaff68acf99573679d registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:1 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:3a8316d42c6491dbc15f2cf59985901365c7c402d3a0649abe695a4a6554654e registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]"
time=2026-10-02T04:04:13.290Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:04:13.437Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 namespace=maestro
time=2026-10-02T04:04:13.594Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 dryRun=false
time=2026-10-02T04:04:13.594Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:04:13.638Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1
time=2026-10-02T04:09:17.935Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-10-02T04:09:17.936Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro-agent not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/management/deploy (stamp=1)2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #63642105865001846706176
PR #71052105860601363304448
alert [svc] FrontendPathLatency fired
full failure pattern:
alert [svc] FrontendPathLatency fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
alert11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-03T00:19:29Z
  Ended:       2026-10-03T00:25:28Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9271680-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9271680-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-03T00:35:30Z
  Ended:       2026-10-03T00:41:31Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9271680-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9271680-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73302106157312019271680
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 26: 1 · Sep 27: 8 · Sep 28: 1 · Sep 29: 19 · Sep 30: 3 · Oct 1: 5 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gc94n755ktzv/cluster-candidate-5-0-l6z5kw should provision
Unexpected error:
    <*fmt.wrapError | 0xc001032d80>: 
    failed to create HCP cluster cluster-candidate-5-0-l6z5kw: failed waiting for cluster="cluster-candidate-5-0-l6z5kw" in resourcegroup="rg-candidate-5-0-gc94n755ktzv" to finish creating: GET https://rp.j1697920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "name": "af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "status": "Failed",
      "startTime": "2026-10-02T11:49:05.164042474Z",
      "endTime": "2026-10-02T12:18:11.434067357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-gc94n755ktzv/cluster-candidate-5-0-l6z5kw should provision
Unexpected error:
    <*fmt.wrapError | 0xc001032d80>: 
    failed to create HCP cluster cluster-candidate-5-0-l6z5kw: failed waiting for cluster="cluster-candidate-5-0-l6z5kw" in resourcegroup="rg-candidate-5-0-gc94n755ktzv" to finish creating: GET https://rp.j1697920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "name": "af1aff57-68d5-48d8-a90a-8e634de5f0cb",
      "status": "Failed",
      "startTime": "2026-10-02T11:49:05.164042474Z",
      "endTime": "2026-10-02T12:18:11.434067357Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73032105975953711697920
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no s...
full failure pattern:
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00156e168>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc00156e168>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxczr.apps.aro.cilium-cluster.5eji.j2884096.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73212106002969722884096
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-c74lpp"
Unexpected error:
    <*fmt.wrapError | 0xc001776360>: 
    DNS for route host agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-c74lpp"
Unexpected error:
    <*fmt.wrapError | 0xc001776360>: 
    DNS for route host agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-f7mjz.apps.aro.p7k6j4s9j1a9f2j.luhb.j0313216.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73052106082635410313216
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000ca0a60>: 
    failed to create HCP cluster ea-listpkp2p7: failed waiting for cluster="ea-listpkp2p7" in resourcegroup="ea-list-4ccm8pbgrf47" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "name": "ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "status": "Failed",
      "startTime": "2026-10-02T15:58:05.107845034Z",
      "endTime": "2026-10-02T16:27:08.62997793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 1m14s ago; version 4.21.36 is Partial (want Completed), started 5m36s ago; version 4.21.36 is Partial (want Completed), started 11m4s ago; version 4.21.36 is Partial (want Completed), started 15m6s ago; version 4.21.36 is Partial (want Completed), started 20m46s ago; version 4.21.36 is Partial (want Completed), started 24m30s ago; version 4.21.36 is Partial (want Completed), started 25m39s ago; version 4.21.36 is Partial (want Completed), started 25m51s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000ca0a60>: 
    failed to create HCP cluster ea-listpkp2p7: failed waiting for cluster="ea-listpkp2p7" in resourcegroup="ea-list-4ccm8pbgrf47" to finish creating: GET https://rp.j4233600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "name": "ff4033f3-93ba-43b3-86a5-c0cc8837ee8a",
      "status": "Failed",
      "startTime": "2026-10-02T15:58:05.107845034Z",
      "endTime": "2026-10-02T16:27:08.62997793Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.36 is Partial (want Completed), started 1m14s ago; version 4.21.36 is Partial (want Completed), started 5m36s ago; version 4.21.36 is Partial (want Completed), started 11m4s ago; version 4.21.36 is Partial (want Completed), started 15m6s ago; version 4.21.36 is Partial (want Completed), started 20m46s ago; version 4.21.36 is Partial (want Completed), started 24m30s ago; version 4.21.36 is Partial (want Completed), started 25m39s ago; version 4.21.36 is Partial (want Completed), started 25m51s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72322106039082894233600
UpdateHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d8120>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-9h5ws9hlzq4p" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-9h5ws9hlzq4p, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*fmt.wrapErrors | 0xc0011d8120>: 
    failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-9h5ws9hlzq4p" to finish updating, caused by: timeout '10.000000' minutes exceeded during UpdateHCPCluster for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-9h5ws9hlzq4p, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892105963382757658624
UpdateHCPCluster20260901 timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster20260901 timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.52%1 of 66 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 0 · Oct 2: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:193]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015c66c0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nqllmtrxwh84" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-nqllmtrxwh84, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:193]: failed to update cluster with new KMS key
Unexpected error:
    <*fmt.wrapErrors | 0xc0015c66c0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-nqllmtrxwh84" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-nqllmtrxwh84, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #68892105963382757658624