CIHealth
<
Single Day: Oct 1▾
Last 7 Days Weekly: Sep 28 - Oct 4 Sprint: Sep 28 - Oct 11 Single Day: Oct 1
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 24 · Failures matched: 75 · Total runs: 86 · Runs affected: 43

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRconsole should return a successful response or redirect, got 503
full failure pattern:
console should return a successful response or redirect, got 503
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e22.33%2 of 86 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 1200.000s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 1200.000s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 600.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:182]: Timed out after 600.002s.
public ingress should be reachable from outside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_public_and_private_kas.go:180 with:
console should return a successful response or redirect, got 503
Expected
    <int>: ...
to be <
    <int>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a default cluster and verify KAS is accessible via both public shared ingress and private Swift networking from worker nodes2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #68072105779586879983616
PR #68072105728864041832448
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRcluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...
full failure pattern:
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.16%1 of 86 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013aaac8>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013aaac8>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72882105612041917042688
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRnode pool upgrade verification failed: <node> (KubeletVersion v1
full failure pattern:
node pool upgrade verification failed: <node> (KubeletVersion v1
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.16%1 of 86 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:205]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc0014b5300>: 
    node pool upgrade verification failed: h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); nodes=[{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":30},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:205]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc0014b5300>: 
    node pool upgrade verification failed: h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg (KubeletVersion v1.33.13 has minor 33, expected 34 for OCP 4.21.35); nodes=[{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-bdpxb","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":30},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-d2h5m","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47},{"name":"h6i8s8g8h0d0f1g-npupgrade-4-20-fq7kg-n4ktg","ready":true,"kubeletVersion":"v1.33.13","containerRuntimeVersion":"cri-o://1.33.13-7.rhaos4.20.gitd6a41c4.el9","imageCount":47}]
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72602105472600422486016
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert66.98%6 of 86 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 25: 6 · Sep 26: 1 · Sep 27: 2 · Sep 28: 1 · Sep 29: 7 · Sep 30: 10 · Oct 1: 6none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T23:46:43Z
  Ended:       2026-10-01T23:52:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1600128-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t69mr53lpaagaj8ndpfbdmi4pj01tkd-j7t7p3j9q1q0x2g", pod="machine-approver-74799c449d-qwlv2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t69mr53lpaagaj8ndpfbdmi4pj01tkd-j7t7p3j9q1q0x2g/machine-approver-74799c449d-qwlv2 has been in a non-ready state for longer than 5 minutes.
alert fired 96 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-mwtcx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-mwtcx has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-vvwct", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-vvwct has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-7bbsw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-7bbsw has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-v4ftx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-v4ftx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zgtdx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zgtdx has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-42mfl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-42mfl has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-lvnnt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-lvnnt has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-pghsk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-pghsk has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gxkml", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gxkml has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-jlzrr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-jlzrr has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-lvjnb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-lvjnb has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-4jj9w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-4jj9w has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-fpstv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-fpstv has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cgnq6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cgnq6 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-gxr4d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-gxr4d has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-98xtm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-98xtm has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-zc7gh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-zc7gh has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-qjtjc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-qjtjc has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-7dl97", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-7dl97 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-msjxj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-msjxj has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-lcvdj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-lcvdj has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cbjnh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cbjnh has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-wbsrk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-wbsrk has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-88hwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-88hwc has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-vpc4x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-vpc4x has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-rm7bz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-rm7bz has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-frwsn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-frwsn has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bmpv2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bmpv2 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-psq6v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-psq6v has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-59fkb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-59fkb has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-s9hxt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-s9hxt has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-d5ggn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-d5ggn has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-mz8hk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-mz8hk has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-rps7j", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-rps7j has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gf894", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gf894 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-9zvzb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-9zvzb has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-29d68", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-29d68 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-g8db9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-g8db9 has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gn9th", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gn9th has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-4pklx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-4pklx has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-f8zbt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-f8zbt has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-qlxtj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-qlxtj has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zrnlh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zrnlh has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-6m7wq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-6m7wq has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-9rhtk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-9rhtk has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bl4jg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bl4jg has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-8pmcr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-8pmcr has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-9nxwc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-9nxwc has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-9f96s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-9f96s has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-mmp68", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-mmp68 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-d4nqn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-d4nqn has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-c5nnh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-c5nnh has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-t48z7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-t48z7 has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-q4bwx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-q4bwx has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-f9m5n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-f9m5n has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-xb9s5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-xb9s5 has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-jh25x", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-jh25x has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-01T18:07:41Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-g9tw9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-g9tw9 has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-r6qng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-r6qng has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-74wcq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-74wcq has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-7lct6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-7lct6 has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-01T18:08:38Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zkbkk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zkbkk has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-v7m2p", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-v7m2p has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-dh2wj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-dh2wj has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-9frwj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-9frwj has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bvxz5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bvxz5 has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-hfm7f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-hfm7f has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-wjp8n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-wjp8n has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-n5vmn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-n5vmn has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-d8lx7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-d8lx7 has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-dlhhb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-dlhhb has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-k69bw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-k69bw has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-kczlx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-kczlx has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-zdgk7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-zdgk7 has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-rvmlf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-rvmlf has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-gmfc6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-gmfc6 has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-lddj8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-lddj8 has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-xm6fg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-xm6fg has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-fn46n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-fn46n has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-58r86", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-58r86 has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:34:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-cfqmp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-cfqmp has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cpccl", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cpccl has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-th4tc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-th4tc has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-2xsng", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-2xsng has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-z8ldz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-z8ldz has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-96jbc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-96jbc has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-cltv6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-cltv6 has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-sfd6l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-sfd6l has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-bgmnc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-bgmnc has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-phv2m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-phv2m has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-2jlkd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-2jlkd has been in a non-ready state for longer than 5 minutes.
Firing 92:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:33:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-cf2v5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-cf2v5 has been in a non-ready state for longer than 5 minutes.
Firing 93:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-gmspn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-gmspn has been in a non-ready state for longer than 5 minutes.
Firing 94:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-7rkp9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-7rkp9 has been in a non-ready state for longer than 5 minutes.
Firing 95:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:28:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y", pod="ignition-server-proxy-7fb844bf7d-2566w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v6276f370teuh06rg23hq77p5h0s-l2e2a6g2r9h9h0y/ignition-server-proxy-7fb844bf7d-2566w has been in a non-ready state for longer than 5 minutes.
Firing 96:
  State:       Resolved
  Started:     2026-10-01T18:08:39Z
  Ended:       2026-10-01T18:30:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5666176-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f", pod="kube-controller-manager-7cfc7cc4bb-66jst", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t64v3u9bus0muojo3trqcut8bp50jk4-v8l6m9c7b8s7p9f/kube-controller-manager-7cfc7cc4bb-66jst has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:11:06Z
  Ended:       2026-10-01T12:17:05Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8197888-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5vg8dltnlcbk8c17tn7tb3mpjhgjq9-u5u0k2w2t5m6a1t", pod="openshift-apiserver-58f5558dc-lmkqq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5vg8dltnlcbk8c17tn7tb3mpjhgjq9-u5u0k2w2t5m6a1t/openshift-apiserver-58f5558dc-lmkqq has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #63642105783166311600128
PR #71622105700267935666176
PR #72762105608440268197888
PR #72662105594300950646784
PR #72832105536022291943424
PR #72762105460820702924800
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert55.81%5 of 86 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 25: 12 · Sep 26: 0 · Sep 27: 2 · Sep 28: 7 · Sep 29: 11 · Sep 30: 3 · Oct 1: 5none
Full failure examples (3)
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T23:10:36Z
  Ended:       2026-10-01T23:23:39Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9983616-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T23:11:38Z
  Ended:       2026-10-01T23:25:38Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9983616-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-10-01T23:12:38Z
  Ended:       2026-10-01T23:23:39Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9983616-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T14:26:18Z
  Ended:       2026-10-01T14:44:16Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4147456-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T11:09:02Z
  Ended:       2026-10-01T11:15:01Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8313344-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #68072105779586879983616
PR #72762105647975744147456
PR #69002105599650848313344
PR #71812105473667151433728
PR #72722105468689062039552
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert55.81%5 of 86 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 25: 3 · Sep 26: 1 · Sep 27: 1 · Sep 28: 1 · Sep 29: 5 · Sep 30: 2 · Oct 1: 5none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T21:56:38Z
  Ended:       2026-10-01T22:24:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T21:56:38Z
  Ended:       2026-10-01T22:24:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:21:23Z
  Ended:       2026-10-01T18:49:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qpxgq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T18:32:21Z
  Ended:       2026-10-01T18:49:23Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qpxgq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:19:00Z
  Ended:       2026-10-01T12:45:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T12:19:00Z
  Ended:       2026-10-01T12:45:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #69242105758317698617344
PR #69002105707780064153600
PR #71582105614591059824640
PR #69002105599650848313344
PR #71812105596839066603520
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert55.81%5 of 86 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 25: 2 · Sep 26: 1 · Sep 27: 0 · Sep 28: 5 · Sep 29: 3 · Sep 30: 4 · Oct 1: 5none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T22:11:26Z
  Ended:       2026-10-01T22:26:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T22:11:26Z
  Ended:       2026-10-01T22:26:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gfv2r", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:36:27Z
  Ended:       2026-10-01T18:50:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qpxgq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:32:56Z
  Ended:       2026-10-01T12:47:55Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-01T12:32:56Z
  Ended:       2026-10-01T12:46:56Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mrs5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #69242105758317698617344
PR #69002105707780064153600
PR #71582105614591059824640
PR #69002105599650848313344
PR #71812105596839066603520
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert55.81%5 of 86 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 1 · Sep 29: 2 · Sep 30: 3 · Oct 1: 5none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T21:59:25Z
  Ended:       2026-10-01T22:29:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a69f5d7-6f9f-4450-a9f7-f182ad9a23c6"
  Description: Pod ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T21:59:25Z
  Ended:       2026-10-01T22:29:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8617344-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.4:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8617344/providers/microsoft.monitor/accounts/hcps-j8617344", namespace="ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a69f5d7-6f9f-4450-a9f7-f182ad9a23c6"
  Description: Pod ocm-arohcpci01-2t6898csqijj9hrgrrv2thav2ags7h5e-u0x9h1i4c5m9r3t/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T18:23:21Z
  Ended:       2026-10-01T18:53:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="71bdc953-4c3d-4f04-a433-73b33d9edd12"
  Description: Pod ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T18:23:21Z
  Ended:       2026-10-01T18:53:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4153600-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4153600/providers/microsoft.monitor/accounts/hcps-j4153600", namespace="ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="71bdc953-4c3d-4f04-a433-73b33d9edd12"
  Description: Pod ocm-arohcpci01-2t6540sfbs5tnji617qi625vcboi5s6g-v2e1p4r9r2e9x7a/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T12:20:58Z
  Ended:       2026-10-01T12:50:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f591f195-5e0c-485f-a44a-f0bfd5b23bc7"
  Description: Pod ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-01T12:20:58Z
  Ended:       2026-10-01T12:49:55Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j9824640-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.94:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9824640/providers/microsoft.monitor/accounts/hcps-j9824640", namespace="ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f591f195-5e0c-485f-a44a-f0bfd5b23bc7"
  Description: Pod ocm-arohcpci01-2t5vpkfr8mo8mfg13tg03gjq59s5mg0f-d4e3m6p6t4b1r3d/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #69242105758317698617344
PR #69002105707780064153600
PR #71582105614591059824640
PR #69002105599650848313344
PR #71812105596839066603520
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e1618.60%16 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 25: 6 · Sep 26: 2 · Sep 27: 7 · Sep 28: 19 · Sep 29: 10 · Sep 30: 8 · Oct 1: 21INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013bd140>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xj578sgpwt6l" to finish creating: GET https://rp.j2660096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "name": "0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "status": "Failed",
      "startTime": "2026-10-02T00:27:55.227836447Z",
      "endTime": "2026-10-02T00:47:04.606186902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013bd140>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-xj578sgpwt6l" to finish creating: GET https://rp.j2660096.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "name": "0fba71ef-f7ca-46bc-a499-4c95fff82d8b",
      "status": "Failed",
      "startTime": "2026-10-02T00:27:55.227836447Z",
      "endTime": "2026-10-02T00:47:04.606186902Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-xj578sgpwt6l--managed/providers/Microsoft.Compute/virtualMachines/u9v6s2e4a2b5l3l-autoscale-noaz-f87v6-bwdd7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:182]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000fb92c0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-d4n779" in resourcegroup="rg-np-version-upgrade-d4n779-msk5sf2n8dpt" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f",
      "name": "d667a581-6512-4aad-a4a8-0592dbc7103f",
      "status": "Failed",
      "startTime": "2026-10-02T00:18:31.430555435Z",
      "endTime": "2026-10-02T00:37:32.349627981Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:182]: failed to create node pool npupgrade-4-21 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000fb92c0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-d4n779" in resourcegroup="rg-np-version-upgrade-d4n779-msk5sf2n8dpt" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d667a581-6512-4aad-a4a8-0592dbc7103f",
      "name": "d667a581-6512-4aad-a4a8-0592dbc7103f",
      "status": "Failed",
      "startTime": "2026-10-02T00:18:31.430555435Z",
      "endTime": "2026-10-02T00:37:32.349627981Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-d4n779-msk5sf2n8dpt-np-upgrade-d4-ed3dd0ec/providers/Microsoft.Compute/virtualMachines/l0u3m1v2v6n0r3x-npupgrade-4-21-bvg9t-b9ms7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq"
Unexpected error:
    <*fmt.wrapError | 0xc0010a7280>: 
    failed to create NodePool np-xxkkqq: failed waiting for nodepool="np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq" in resourcegroup="rg-hs-pre--9kx2vcffdvgg" to finish creating: GET https://rp.j0967040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "name": "9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "status": "Failed",
      "startTime": "2026-10-01T22:20:04.161467493Z",
      "endTime": "2026-10-01T22:39:14.657703099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq"
Unexpected error:
    <*fmt.wrapError | 0xc0010a7280>: 
    failed to create NodePool np-xxkkqq: failed waiting for nodepool="np-xxkkqq" for cluster "cluster-hs-pre-xxkkqq" in resourcegroup="rg-hs-pre--9kx2vcffdvgg" to finish creating: GET https://rp.j0967040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "name": "9b6a2e90-815b-425d-ab8b-0de7336ee353",
      "status": "Failed",
      "startTime": "2026-10-01T22:20:04.161467493Z",
      "endTime": "2026-10-01T22:39:14.657703099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--9kx2vcffdvgg-xxkkqq--managed/providers/Microsoft.Compute/virtualMachines/h3e6a3z6w8m5h2c-np-xxkkqq-2gbgl-78tdt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (17)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor1
Affected runs (16)
Date (UTC)Associated PRProw job
PR #67752105799429062660096
PR #72932105791127218032640
PR #71052105766044650967040
PR #73032105721100032806912
PR #72962105705639958286336
PR #71622105700267935666176
PR #72272105693768928727040
PR #67752105656281044881408
PR #72922105647499015360512
PR #71122105619693527109632
PR #71052105609202608115712
PR #72762105608440268197888
PR #72322105598359803793408
PR #72682105596656899592192
PR #72662105594300950646784
PR #72762105572392540573696
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e55.81%5 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsSep 25: 2 · Sep 26: 1 · Sep 27: 8 · Sep 28: 1 · Sep 29: 19 · Sep 30: 3 · Oct 1: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001422540>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-nr76gskcfsht" to finish creating: GET https://rp.j8617344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "name": "d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "status": "Failed",
      "startTime": "2026-10-01T21:37:37.328442226Z",
      "endTime": "2026-10-01T22:06:41.36782171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001422540>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-nr76gskcfsht" to finish creating: GET https://rp.j8617344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "name": "d63268b6-f286-4ac7-9dcb-f75b1fa2b29a",
      "status": "Failed",
      "startTime": "2026-10-01T21:37:37.328442226Z",
      "endTime": "2026-10-01T22:06:41.36782171Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-dhvd4s" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000ed0460>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-dhvd4s: failed waiting for cluster="cp-ystream-upgrade-5-1-dhvd4s" in resourcegroup="rg-cp-ystream-upgrade-5-1-25w79q54vqwd" to finish creating: GET https://rp.j4153600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "name": "e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "status": "Failed",
      "startTime": "2026-10-01T18:01:36.572713358Z",
      "endTime": "2026-10-01T18:30:44.899304602Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-dhvd4s" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0xc000ed0460>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-dhvd4s: failed waiting for cluster="cp-ystream-upgrade-5-1-dhvd4s" in resourcegroup="rg-cp-ystream-upgrade-5-1-25w79q54vqwd" to finish creating: GET https://rp.j4153600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "name": "e4f26d35-7dcc-4a11-95e9-ba0b44e92249",
      "status": "Failed",
      "startTime": "2026-10-01T18:01:36.572713358Z",
      "endTime": "2026-10-01T18:30:44.899304602Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-jsgjhmrsczb4/cluster-candidate-4-23-tqz4r8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011a41c0>: 
    failed to create HCP cluster cluster-candidate-4-23-tqz4r8: failed waiting for cluster="cluster-candidate-4-23-tqz4r8" in resourcegroup="rg-candidate-4-23-jsgjhmrsczb4" to finish creating: GET https://rp.j9824640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "name": "3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "status": "Failed",
      "startTime": "2026-10-01T11:58:08.343305291Z",
      "endTime": "2026-10-01T12:27:12.911892104Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-jsgjhmrsczb4/cluster-candidate-4-23-tqz4r8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0011a41c0>: 
    failed to create HCP cluster cluster-candidate-4-23-tqz4r8: failed waiting for cluster="cluster-candidate-4-23-tqz4r8" in resourcegroup="rg-candidate-4-23-jsgjhmrsczb4" to finish creating: GET https://rp.j9824640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "name": "3b3b1244-2be2-4c28-a2e6-54d138ae923b",
      "status": "Failed",
      "startTime": "2026-10-01T11:58:08.343305291Z",
      "endTime": "2026-10-01T12:27:12.911892104Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (5)
Date (UTC)Associated PRProw job
PR #69242105758317698617344
PR #69002105707780064153600
PR #71582105614591059824640
PR #69002105599650848313344
PR #71812105596839066603520
alert [svc] KustoLogsDataStale fired
full failure pattern:
alert [svc] KustoLogsDataStale fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
alert33.49%3 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 2 · Sep 30: 2 · Oct 1: 3none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T19:40:45Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2640384-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.90:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2640384/providers/microsoft.monitor/accounts/services-j2640384", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-77d869db9f-wr2wh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2640384-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T19:40:45Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j2640384-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.90:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2640384/providers/microsoft.monitor/accounts/services-j2640384", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-77d869db9f-wr2wh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j2640384-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T15:34:25Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4147456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.82:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4147456/providers/microsoft.monitor/accounts/services-j4147456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-9976f6db4-kdw9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4147456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T15:35:24Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j4147456-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.82:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4147456/providers/microsoft.monitor/accounts/services-j4147456", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-9976f6db4-kdw9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j4147456-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T14:24:02Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5027584-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5027584/providers/microsoft.monitor/accounts/services-j5027584", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-65fddd48b7-ckqvd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5027584-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T14:24:02Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j5027584-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.77:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5027584/providers/microsoft.monitor/accounts/services-j5027584", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-65fddd48b7-ckqvd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j5027584-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #72922105711608922640384
PR #72762105647975744147456
PR #70552105632268105027584
Command failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-<id>/pipeline@sha256:<hex> arohcpsvcdev.azurecr.io/ci-op-<id>/pipeline:<hex> --from-registry-confi...
full failure pattern:
Command failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-<id>/pipeline@sha256:<hex> arohcpsvcdev.azurecr.io/ci-op-<id>/pipeline:<hex> --from-registry-config /tmp/tmp.<id>/containers/auth.json --to-registry-config
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision22.33%2 of 86 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 2none
Full failure examples (2)
time=2026-10-01T15:34:21.055Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.MgmtAgent resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> mgmtAgent.image.registry <nil>}:{<nil> mgmtAgent.image.repository <nil>}@{<nil> mgmtAgent.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-01T15:35:58.894Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.MgmtAgent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.RKxD35P2Nz/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-8mw8v43n/pipeline@sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b to arohcpsvcdev.azurecr.io/ci-op-8mw8v43n/pipeline:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b.\nThe image will still be available under it's original digest sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b in the target registry.\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-8mw8v43n/pipeline@sha256:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b arohcpsvcdev.azurecr.io/ci-op-8mw8v43n/pipeline:8d86cc812124ab7af5c3f581f9f2865d36cb1fd9340ecc28481c6d525c0cfa0b --from-registry-config /tmp/tmp.RKxD35P2Nz/containers/auth.json --to-registry-config /tmp/tmp.RKxD35P2Nz/containers/auth.json\n exit status 1"
time=2026-10-01T15:33:44.701Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> fleet.image.registry <nil>}:{<nil> fleet.image.repository <nil>}@{<nil> fleet.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-10-01T15:35:32.216Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build11.ci.openshift.org\nSaved credentials for registry.build11.ci.openshift.org into /tmp/tmp.WQPJJsgkPI/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build11.ci.openshift.org/ci-op-49jbb95d/pipeline@sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 to arohcpsvcdev.azurecr.io/ci-op-49jbb95d/pipeline:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14.\nThe image will still be available under it's original digest sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 in the target registry.\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14: unknown: unknown error: map[DriverName:s3aws Enclosed:map[RequestFailure:map[]]]\nCommand failed after 5 attempts: oras cp registry.build11.ci.openshift.org/ci-op-49jbb95d/pipeline@sha256:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 arohcpsvcdev.azurecr.io/ci-op-49jbb95d/pipeline:0d10cd3c8fa8572aa7784bf13bac7b0907199a7c1851db08138b412e3f25df14 --from-registry-config /tmp/tmp.WQPJJsgkPI/containers/auth.json --to-registry-config /tmp/tmp.WQPJJsgkPI/containers/auth.json\n exit status 1"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Fleet/global/mirror-image1
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.MgmtAgent/global/mirror-image1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69002105670637572853760
PR #72812105670047690133504
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
provision22.33%2 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsSep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 3none
Full failure examples (2)
time=2026-10-01T19:47:54.330Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-01T19:47:56.434Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-01T19:47:57.612Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=7fa4e9b0d5432f3733a8410e6c9eae279686c4d2104771439ec6c49754f75d3c portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2785792-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F7fa4e9b0d5432f3733a8410e6c9eae279686c4d2104771439ec6c49754f75d3c
time=2026-10-01T19:50:58.074Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/7fa4e9b0d5432f3733a8410e6c9eae279686c4d2104771439ec6c49754f75d3c/operationStatuses/08584107228086173802\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/cluster-suocoh45plnvg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/ah-ci01-se-j2785792-1\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n                \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2785792-svc/providers/Microsoft.Resources/deployments/ah-ci01-se-j2785792-1\\\",\\r\\n                \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n                \\\"details\\\": [\\r\\n                  {\\r\\n                    \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n                    \\\"message\\\": \\\"The vault name 'ah-ci01-se-j2785792-1' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n                  }\\r\\n                ]\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-01T19:47:48.734Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 description="Step infra\n  Kind: ARM\n  Template: templates/mgmt-infra.bicep\n  Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-10-01T19:47:49.512Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2
time=2026-10-01T19:47:50.952Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 deployment=5be4ec9a5f0977ac9396f709d63fcfad01e98a314c6ed4279d81457cdf4fb342 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0582272-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5be4ec9a5f0977ac9396f709d63fcfad01e98a314c6ed4279d81457cdf4fb342
time=2026-10-01T19:50:21.739Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/5be4ec9a5f0977ac9396f709d63fcfad01e98a314c6ed4279d81457cdf4fb342/operationStatuses/08584107228152517010\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/ah-ci01-me-j0582272-2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-me-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/mgmt-kv-2awbxe73l63vi\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-mg-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/msi-kv-g2p3wnorghzau\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-mi-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/infra (stamp=2)1
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73062105741109882785792
PR #71432105738861190582272
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e22.33%2 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.40
Unexpected error:
    <*fmt.wrapErrors | 0xc0013de390>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-vvt8wh" resourcegroup="rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-vvt8wh in resource group rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.40
Unexpected error:
    <*fmt.wrapErrors | 0xc0013de390>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-vvt8wh" resourcegroup="rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-vvt8wh in resource group rg-np-version-upgrade-vvt8wh-bgjbrzltjmjv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002898f60>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-mbclvmsdh2bf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-mbclvmsdh2bf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc002898f60>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-mbclvmsdh2bf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-mbclvmsdh2bf, error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72762105647975744147456
PR #72882105612041917042688
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 2 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc00262f770>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.007s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc00262f770>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72272105693768928727040
ERROR CODE: 429; provider Microsoft.ManagedIdentity
full failure pattern:
ERROR CODE: 429; provider Microsoft.ManagedIdentity
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 25: 2 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 1 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:108]: failed to get service managed identity
Unexpected error:
    <*exported.ResponseError | 0xc0017f8e10>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:108]: failed to get service managed identity
Unexpected error:
    <*exported.ResponseError | 0xc0017f8e10>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71432105804050531356672
failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress context deadline exceed...
full failure pattern:
failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
provision11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1 · Oct 1: 1none
Full failure examples (1)
time=2026-10-01T20:26:41.918Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-10-01T20:26:42.226Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4]"
time=2026-10-01T20:26:42.226Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-01T20:26:42.418Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-10-01T20:26:42.591Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-10-01T20:26:42.591Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-01T20:26:42.660Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-10-01T20:31:45.711Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-10-01T20:31:45.712Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71812105746107995787264
ERROR CODE: DeploymentFailed; detail code BadRequest; detail message The requested operation cannot be performed because the database account arohcpci01-rp-j2503040 is in the proc...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code BadRequest; detail message The requested operation cannot be performed because the database account arohcpci01-rp-j2503040 is in the process of being created. Please wait until account creation completes and retry the operation.\\\\\\\ \\\\\\ ActivityId: <uuid>, Microsoft.Azure.Documents.Common/2.14.0"\\\ \\ }; provider Microsoft.DocumentDB
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
time=2026-10-01T19:47:52.399Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-01T19:47:54.301Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-01T19:47:55.661Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=81d379c4f7c29ce92f1cd109cb00ebbe0457eec7d8a38ac495b54ae648addb81 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2503040%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F81d379c4f7c29ce92f1cd109cb00ebbe0457eec7d8a38ac495b54ae648addb81
time=2026-10-01T19:49:26.400Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2503040/providers/Microsoft.Resources/deployments/81d379c4f7c29ce92f1cd109cb00ebbe0457eec7d8a38ac495b54ae648addb81/operationStatuses/08584107228106955756\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2503040/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2503040/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j2503040\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"BadRequest\\\\\\\",\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"The requested operation cannot be performed because the database account arohcpci01-rp-j2503040 is in the process of being created. Please wait until account creation completes and retry the operation.\\\\\\\\r\\\\\\\\nActivityId: deddb421-d755-4f62-b0ea-94301557d50e, Microsoft.Azure.Documents.Common/2.14.0\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67752105739854892503040
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsSep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 3none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:91]: failed to deploy customer infrastructure
Unexpected error:
    <*fmt.wrapError | 0xc0008d79c0>: 
    failed waiting for deployment "customer-infra-delayed-rbac-cluster" in resourcegroup="delayed-rbac-wx4qr5bhxpxx" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delayed-rbac-wx4qr5bhxpxx/providers/Microsoft.Resources/deployments/customer-infra-delayed-rbac-cluster/operationStatuses/08584107303732954713
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-usudlv6oegmxo' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:91]: failed to deploy customer infrastructure
Unexpected error:
    <*fmt.wrapError | 0xc0008d79c0>: 
    failed waiting for deployment "customer-infra-delayed-rbac-cluster" in resourcegroup="delayed-rbac-wx4qr5bhxpxx" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delayed-rbac-wx4qr5bhxpxx/providers/Microsoft.Resources/deployments/customer-infra-delayed-rbac-cluster/operationStatuses/08584107303732954713
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-usudlv6oegmxo' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67752105699867165724672
alert [svc] KubeContainerOOMKilled fired
full failure pattern:
alert [svc] KubeContainerOOMKilled fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
alert11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-01T07:22:46Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j1943424-mgmt-1", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1943424/providers/microsoft.monitor/accounts/services-j1943424", namespace="arobit", pod="arobit-forwarder-b8lv8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fe636689-1a0b-4733-829d-e26d9c7f983a"
  Description: Container fluentbit in pod arobit/arobit-forwarder-b8lv8 on cluster ci01-j1943424-mgmt-1 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-01T07:22:47Z
  Severity:    Sev3
  Labels:      alertname="KubeContainerOOMKilled", cluster="ci01-j1943424-mgmt-1", component="kubernetes-infrastructure", container="fluentbit", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1943424/providers/microsoft.monitor/accounts/services-j1943424", namespace="arobit", pod="arobit-forwarder-b8lv8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="OOMKilled", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fe636689-1a0b-4733-829d-e26d9c7f983a"
  Description: Container fluentbit in pod arobit/arobit-forwarder-b8lv8 on cluster ci01-j1943424-mgmt-1 has been OOMKilled. This indicates the container exceeded its memory limit and was terminated by the kernel.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeContainerOOMKilled does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72832105536022291943424
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000739be0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-v5pj4qqth2gl" to finish creating: GET https://rp.j3797248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca",
      "name": "523b7cc8-b248-458d-8692-3a89ce945aca",
      "status": "Failed",
      "startTime": "2026-10-01T13:50:45.474668418Z",
      "endTime": "2026-10-01T14:09:46.073738279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000739be0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-v5pj4qqth2gl" to finish creating: GET https://rp.j3797248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/523b7cc8-b248-458d-8692-3a89ce945aca",
      "name": "523b7cc8-b248-458d-8692-3a89ce945aca",
      "status": "Failed",
      "startTime": "2026-10-01T13:50:45.474668418Z",
      "endTime": "2026-10-01T14:09:46.073738279Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-v5pj4qqth2gl--managed/providers/Microsoft.Compute/virtualMachines/n7b6g4l4l8u2c3w-np-deletable-nlxsl-g2gld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72662105632583583797248
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-tqsp5m"
Unexpected error:
    <*fmt.wrapError | 0xc0016712e0>: 
    DNS for route host agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-tqsp5m"
Unexpected error:
    <*fmt.wrapError | 0xc0016712e0>: 
    DNS for route host agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4thdf.apps.aro.l4o4l3o0x3x9d8p.z2rt.j3793408.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72322105598359803793408
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:130]: HCP cluster rg-candidate-4-20-vgwv7thnmvhb/cluster-candidate-4-20-mvn65b should provision
Unexpected error:
    <*fmt.wrapError | 0xc001036300>: 
    failed to create HCP cluster cluster-candidate-4-20-mvn65b: failed waiting for cluster="cluster-candidate-4-20-mvn65b" in resourcegroup="rg-candidate-4-20-vgwv7thnmvhb" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "name": "c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "status": "Failed",
      "startTime": "2026-10-01T23:41:46.834189313Z",
      "endTime": "2026-10-02T00:10:52.697527079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.40 is Partial (want Completed), started 1m39s ago; version 4.20.40 is Partial (want Completed), started 4m49s ago; version 4.20.40 is Partial (want Completed), started 10m52s ago; version 4.20.40 is Partial (want Completed), started 14m44s ago; version 4.20.40 is Partial (want Completed), started 19m16s ago; version 4.20.40 is Partial (want Completed), started 21m37s ago; version 4.20.40 is Partial (want Completed), started 23m41s ago; version 4.20.40 is Partial (want Completed), started 24m16s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:130]: HCP cluster rg-candidate-4-20-vgwv7thnmvhb/cluster-candidate-4-20-mvn65b should provision
Unexpected error:
    <*fmt.wrapError | 0xc001036300>: 
    failed to create HCP cluster cluster-candidate-4-20-mvn65b: failed waiting for cluster="cluster-candidate-4-20-mvn65b" in resourcegroup="rg-candidate-4-20-vgwv7thnmvhb" to finish creating: GET https://rp.j8032640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "name": "c8df1e89-0fd9-4b56-a005-77ef918b3bfb",
      "status": "Failed",
      "startTime": "2026-10-01T23:41:46.834189313Z",
      "endTime": "2026-10-02T00:10:52.697527079Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.40 is Partial (want Completed), started 1m39s ago; version 4.20.40 is Partial (want Completed), started 4m49s ago; version 4.20.40 is Partial (want Completed), started 10m52s ago; version 4.20.40 is Partial (want Completed), started 14m44s ago; version 4.20.40 is Partial (want Completed), started 19m16s ago; version 4.20.40 is Partial (want Completed), started 21m37s ago; version 4.20.40 is Partial (want Completed), started 23m41s ago; version 4.20.40 is Partial (want Completed), started 24m16s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72932105791127218032640
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...
full failure pattern:
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.16%1 of 86 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002258720>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc002258720>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-n7r7f.apps.aro.cilium-cl.ni9g.j7451520.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73142105808059497451520

Environment: INT

Failure patterns: 3 · Failures matched: 34 · Total runs: 6 · Runs affected: 4

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value canno...
full failure pattern:
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e350.00%3 of 6 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 10 · Oct 1: 30PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-fpkhgg" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0013695c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-x2qm64ntnf4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-fpkhgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-fpkhgg" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0013695c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-x2qm64ntnf4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-fpkhgg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5lppxg" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c77c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-9qkw2hc8j64k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5lppxg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-5lppxg" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000c77c20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-9qkw2hc8j64k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5lppxg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5jgcvl" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00156e510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-qqh9mh2dgv5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5jgcvl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:156]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-5jgcvl" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00156e510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-qqh9mh2dgv5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5jgcvl
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (10)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.213
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.223
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.233
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.03
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2105606654199336960
n/a2105558232767401984
n/a2105471177496465408
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e233.33%2 of 6 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 1 · Sep 29: 4 · Sep 30: 2 · Oct 1: 3DEV, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012c9e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fxmrv96q875w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269",
      "name": "a1048b90-244a-4c5a-ba89-e1135a68a269",
      "status": "Failed",
      "startTime": "2026-10-01T20:23:38.274618399Z",
      "endTime": "2026-10-01T20:42:38.037012934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012c9e80>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fxmrv96q875w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a1048b90-244a-4c5a-ba89-e1135a68a269",
      "name": "a1048b90-244a-4c5a-ba89-e1135a68a269",
      "status": "Failed",
      "startTime": "2026-10-01T20:23:38.274618399Z",
      "endTime": "2026-10-01T20:42:38.037012934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-26bmw to exist\nMachine y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fxmrv96q875w--managed/providers/Microsoft.Compute/virtualMachines/y7y7b3o0h7v8m9j-np-1-tdqpw-dq6sk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk"
Unexpected error:
    <*fmt.wrapError | 0xc000941160>: 
    failed to create NodePool np-jv9hvk: failed waiting for nodepool="np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk" in resourcegroup="rg-candidate-4-20-6nfp6mj4gt68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "name": "29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "status": "Failed",
      "startTime": "2026-10-01T20:46:24.926645724Z",
      "endTime": "2026-10-01T21:05:28.967720321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk"
Unexpected error:
    <*fmt.wrapError | 0xc000941160>: 
    failed to create NodePool np-jv9hvk: failed waiting for nodepool="np-jv9hvk" for cluster "cluster-candidate-4-20-jv9hvk" in resourcegroup="rg-candidate-4-20-6nfp6mj4gt68" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "name": "29d69384-8f10-43a3-adab-ad5ed046a0d7",
      "status": "Failed",
      "startTime": "2026-10-01T20:46:24.926645724Z",
      "endTime": "2026-10-01T21:05:28.967720321Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6nfp6mj4gt68-candidate-jv9hvk--managed/providers/Microsoft.Compute/virtualMachines/f9f2c2u0c2w2q1m-np-jv9hvk-rcvg7-q6krr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00057dcc0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jspphct49kmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb",
      "name": "4a191606-d1f9-418e-b590-0307f5435adb",
      "status": "Failed",
      "startTime": "2026-10-01T01:58:13.545499376Z",
      "endTime": "2026-10-01T02:17:18.498756095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc00057dcc0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jspphct49kmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a191606-d1f9-418e-b590-0307f5435adb",
      "name": "4a191606-d1f9-418e-b590-0307f5435adb",
      "status": "Failed",
      "startTime": "2026-10-01T01:58:13.545499376Z",
      "endTime": "2026-10-01T02:17:18.498756095Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jspphct49kmw--managed/providers/Microsoft.Compute/virtualMachines/x9l9x8u4y2b5f7r-arm64-vm-np-1-lt5pz-7ckxj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105746340389588992
n/a2105471177496465408
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e116.67%1 of 6 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 25: 0 · Sep 26: 1 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00109d0c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s7p6hptqhbvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1",
      "name": "357a31b9-a87e-4b62-a8b5-196b066234f1",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:38.669148771Z",
      "endTime": "2026-10-01T03:17:44.618360897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00109d0c0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-s7p6hptqhbvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/357a31b9-a87e-4b62-a8b5-196b066234f1",
      "name": "357a31b9-a87e-4b62-a8b5-196b066234f1",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:38.669148771Z",
      "endTime": "2026-10-01T03:17:44.618360897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m5a5s2c3g7w1p0l-autoscale-noaz-8xp74-xzdr6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105471177496465408

Environment: PROD

Failure patterns: 9 · Failures matched: 33 · Total runs: 12 · Runs affected: 6

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e18.33%1 of 12 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 25: 5 · Sep 26: 5 · Sep 27: 0 · Sep 28: 5 · Sep 29: 5 · Sep 30: 0 · Oct 1: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000f4b7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-5rjr485pnvns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000f4b7a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-5rjr485pnvns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0013b7a10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-55xcrpmcpjd5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc0013b7a10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-55xcrpmcpjd5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000a6f140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwrdblk56ldw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000a6f140>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwrdblk56ldw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105795013534289920
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e18.33%1 of 12 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 25: 4 · Sep 26: 2 · Sep 27: 4 · Sep 28: 4 · Sep 29: 4 · Sep 30: 4 · Oct 1: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc000ed4500>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ngvxnx" in resourcegroup="rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ngvxnx/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-10-01-125400
Unexpected error:
    <*fmt.wrapError | 0xc000ed4500>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ngvxnx" in resourcegroup="rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ngvxnx-ztnvkxg4zlcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ngvxnx/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq"
Unexpected error:
    <*fmt.wrapError | 0xc0012f2520>: 
    failed to create NodePool np-ks75xq: failed starting nodepool creation "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" in resourcegroup="rg-nightly-4-20-2zfvw4gzpm77": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2zfvw4gzpm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-ks75xq/nodePools/np-ks75xq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq"
Unexpected error:
    <*fmt.wrapError | 0xc0012f2520>: 
    failed to create NodePool np-ks75xq: failed starting nodepool creation "np-ks75xq" for cluster "cluster-nightly-4-20-ks75xq" in resourcegroup="rg-nightly-4-20-2zfvw4gzpm77": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-2zfvw4gzpm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-ks75xq/nodePools/np-ks75xq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-10-01-125400\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105795013534289920
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e18.33%1 of 12 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1 · Sep 29: 3 · Sep 30: 2 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014996b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-10-01-083859, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.008s.
Expected success, but got an error:
    <*errors.joinError | 0xc0014996b0>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=4.21.0-0.nightly-multi-2026-10-01-083859, targetMinor=4.22.0-0.nightly-multi-2026-07-08-192436) failed: clusterversion status.history has no version in target minor "4.22.0-0.nightly-multi-2026-07-08-192436"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105795013534289920
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history
e2e433.33%4 of 12 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4 · Sep 29: 10 · Sep 30: 2 · Oct 1: 6DEV, INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001797968>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7fp2gpvrmpv5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc",
      "name": "fbace7f5-c615-4f66-9959-02331d194dcc",
      "status": "Failed",
      "startTime": "2026-10-01T23:17:05.438751852Z",
      "endTime": "2026-10-01T23:36:05.105086237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001797968>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7fp2gpvrmpv5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fbace7f5-c615-4f66-9959-02331d194dcc",
      "name": "fbace7f5-c615-4f66-9959-02331d194dcc",
      "status": "Failed",
      "startTime": "2026-10-01T23:17:05.438751852Z",
      "endTime": "2026-10-01T23:36:05.105086237Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-74m7b-t7cjp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7fp2gpvrmpv5--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-74m7b-t7cjp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cbe8e0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-f85th5" in resourcegroup="rg-np-noedge-f85th5-2lj4rn8xdzrh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "name": "3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "status": "Failed",
      "startTime": "2026-10-01T18:02:34.984361176Z",
      "endTime": "2026-10-01T18:21:38.6870989Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000cbe8e0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-f85th5" in resourcegroup="rg-np-noedge-f85th5-2lj4rn8xdzrh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "name": "3cf330e8-6cc6-4d77-bd07-24fe8abc4160",
      "status": "Failed",
      "startTime": "2026-10-01T18:02:34.984361176Z",
      "endTime": "2026-10-01T18:21:38.6870989Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-f85th5-2lj4rn8xdzrh-np-ne-f85th5--managed/providers/Microsoft.Compute/virtualMachines/g4z7c0q0c5o8b3k-npnoedge-4-20-fvppx-vw46h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0010769c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g5k9xrk5kbzf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680",
      "name": "1864d250-37c2-44c1-bbbf-426f923fe680",
      "status": "Failed",
      "startTime": "2026-10-01T16:25:59.009326346Z",
      "endTime": "2026-10-01T16:45:03.114189778Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc0010769c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-g5k9xrk5kbzf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/1864d250-37c2-44c1-bbbf-426f923fe680",
      "name": "1864d250-37c2-44c1-bbbf-426f923fe680",
      "status": "Failed",
      "startTime": "2026-10-01T16:25:59.009326346Z",
      "endTime": "2026-10-01T16:45:03.114189778Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-g5k9xrk5kbzf--managed/providers/Microsoft.Compute/virtualMachines/m2q7s0u7z1v5z9m-arm64-vm-np-1-z7spz-94nqz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2105795013534289920
n/a2105687094121730048
n/a2105520036851486720
n/a2105518122772467712
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or q...
full failure pattern:
selector "<selector>" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e216.67%2 of 12 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 1 · Sep 30: 0 · Oct 1: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001f80160>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc001f80160>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc0012cb640>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:88]: failed to select a VM size with ephemeral OS disk support; this typically indicates a SKU restriction or quota issue in the test subscription/region
Unexpected error:
    <*fmt.wrapError | 0xc0012cb640>: 
    selector "ephemeral-osdisk-worker" matched no usable VM size in eastus2: no unrestricted VM size satisfied the selector in the test location; this typically indicates a VM SKU restriction or quota issue with the test subscription/region, not a product defect
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105687094121730048
n/a2105518122772467712
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value canno...
full failure pattern:
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e18.33%1 of 12 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 10INT, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4mwqz2" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00144a120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jlsd79zkk6hh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4mwqz2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-4mwqz2" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00144a120>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-jlsd79zkk6hh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-4mwqz2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-4fwf7q" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001226630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wnp2jmpdpwln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4fwf7q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-4fwf7q" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc001226630>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-wnp2jmpdpwln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4fwf7q
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-24sw5t" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000cdd5c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vhz5sh8l55g9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-24sw5t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-24sw5t" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc000cdd5c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vhz5sh8l55g9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-24sw5t
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (10)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105477920750309376
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet po...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e18.33%1 of 12 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 0 · Sep 30: 0 · Oct 1: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-47sckvjgzdjp/cluster-nightly-4-22-wwdkq6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d874a0>: 
    failed to create HCP cluster cluster-nightly-4-22-wwdkq6: failed waiting for cluster="cluster-nightly-4-22-wwdkq6" in resourcegroup="rg-nightly-4-22-47sckvjgzdjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "name": "4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "status": "Failed",
      "startTime": "2026-10-01T23:33:29.768708196Z",
      "endTime": "2026-10-02T00:02:35.156751805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-47sckvjgzdjp/cluster-nightly-4-22-wwdkq6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d874a0>: 
    failed to create HCP cluster cluster-nightly-4-22-wwdkq6: failed waiting for cluster="cluster-nightly-4-22-wwdkq6" in resourcegroup="rg-nightly-4-22-47sckvjgzdjp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "name": "4b93cf23-0f8f-4a86-986d-b583c88b0cd9",
      "status": "Failed",
      "startTime": "2026-10-01T23:33:29.768708196Z",
      "endTime": "2026-10-02T00:02:35.156751805Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-szptjh" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000f34e40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-szptjh: failed waiting for cluster="cp-ystream-upgrade-4-23-szptjh" in resourcegroup="rg-cp-ystream-upgrade-4-23-46pm69d84hvp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "name": "780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "status": "Failed",
      "startTime": "2026-10-01T23:37:58.369447145Z",
      "endTime": "2026-10-02T00:06:59.154886423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-szptjh" at install version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc000f34e40>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-szptjh: failed waiting for cluster="cp-ystream-upgrade-4-23-szptjh" in resourcegroup="rg-cp-ystream-upgrade-4-23-46pm69d84hvp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "name": "780e4e8f-ebe0-42ed-942e-9333a940dc3c",
      "status": "Failed",
      "startTime": "2026-10-01T23:37:58.369447145Z",
      "endTime": "2026-10-02T00:06:59.154886423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-bjmvhq at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0010869e0>: 
    failed to create HCP cluster np-skip-minor-bjmvhq: failed waiting for cluster="np-skip-minor-bjmvhq" in resourcegroup="rg-np-skip-minor-bjmvhq-v7kckbmqz7jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3",
      "name": "94714044-9af3-4f81-b892-f8e47c0efbb3",
      "status": "Failed",
      "startTime": "2026-10-02T00:05:39.688922023Z",
      "endTime": "2026-10-02T00:34:39.296960872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-bjmvhq at version 4.22.0-0.nightly-multi-2026-07-08-192436
Unexpected error:
    <*fmt.wrapError | 0xc0010869e0>: 
    failed to create HCP cluster np-skip-minor-bjmvhq: failed waiting for cluster="np-skip-minor-bjmvhq" in resourcegroup="rg-np-skip-minor-bjmvhq-v7kckbmqz7jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/94714044-9af3-4f81-b892-f8e47c0efbb3",
      "name": "94714044-9af3-4f81-b892-f8e47c0efbb3",
      "status": "Failed",
      "startTime": "2026-10-02T00:05:39.688922023Z",
      "endTime": "2026-10-02T00:34:39.296960872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105795013534289920
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e18.33%1 of 12 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000638600>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2nzmgkxwzltg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535",
      "name": "eaa1d5d5-fc6a-4816-a207-694d27453535",
      "status": "Failed",
      "startTime": "2026-10-01T05:17:35.937388036Z",
      "endTime": "2026-10-01T05:36:39.648087048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000638600>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-2nzmgkxwzltg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/eaa1d5d5-fc6a-4816-a207-694d27453535",
      "name": "eaa1d5d5-fc6a-4816-a207-694d27453535",
      "status": "Failed",
      "startTime": "2026-10-01T05:17:35.937388036Z",
      "endTime": "2026-10-01T05:36:39.648087048Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2s0w6f2w1a9o6l-np-1-wxf5k-nvwd8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105520443560562688
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e18.33%1 of 12 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001802258>: 
    resource group "rg-zstream-upgrade-4-20-26f5nw6rjl84": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-m4xqll" in resourcegroup="rg-zstream-upgrade-4-20-26f5nw6rjl84" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "name": "fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "status": "Failed",
      "startTime": "2026-10-01T17:28:43.347525555Z",
      "endTime": "2026-10-01T17:52:46.138429989Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t648pvv6ett9l7cgmh574bcs7i8oe4q still exists (deletion dispatched at 2026-10-01T17:28:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001802258>: 
    resource group "rg-zstream-upgrade-4-20-26f5nw6rjl84": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-m4xqll" in resourcegroup="rg-zstream-upgrade-4-20-26f5nw6rjl84" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "name": "fdb2e512-dda1-4736-993b-cb6fd011fea5",
      "status": "Failed",
      "startTime": "2026-10-01T17:28:43.347525555Z",
      "endTime": "2026-10-01T17:52:46.138429989Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t648pvv6ett9l7cgmh574bcs7i8oe4q still exists (deletion dispatched at 2026-10-01T17:28:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105687094121730048

Environment: STG

Failure patterns: 6 · Failures matched: 27 · Total runs: 4 · Runs affected: 3

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
e2e375.00%3 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsSep 25: 0 · Sep 26: 17 · Sep 27: 1 · Sep 28: 2 · Sep 29: 1 · Sep 30: 10 · Oct 1: 6DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f50a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-6svllmjvpbpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b",
      "name": "7c5e553a-2bca-431a-819c-94122ad78b6b",
      "status": "Failed",
      "startTime": "2026-10-02T00:04:50.703195637Z",
      "endTime": "2026-10-02T00:23:53.181881109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f50a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-6svllmjvpbpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b",
      "name": "7c5e553a-2bca-431a-819c-94122ad78b6b",
      "status": "Failed",
      "startTime": "2026-10-02T00:04:50.703195637Z",
      "endTime": "2026-10-02T00:23:53.181881109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k96n75" for cluster "cluster-candidate-5-0-k96n75"
Unexpected error:
    <*fmt.wrapError | 0xc000a16440>: 
    failed to create NodePool np-k96n75: failed waiting for nodepool="np-k96n75" for cluster "cluster-candidate-5-0-k96n75" in resourcegroup="rg-candidate-5-0-gclccxznszk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "name": "e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "status": "Failed",
      "startTime": "2026-10-01T23:23:36.842890739Z",
      "endTime": "2026-10-01T23:42:37.244899482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: Creating: virtualmachine creating or updating\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k96n75" for cluster "cluster-candidate-5-0-k96n75"
Unexpected error:
    <*fmt.wrapError | 0xc000a16440>: 
    failed to create NodePool np-k96n75: failed waiting for nodepool="np-k96n75" for cluster "cluster-candidate-5-0-k96n75" in resourcegroup="rg-candidate-5-0-gclccxznszk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "name": "e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "status": "Failed",
      "startTime": "2026-10-01T23:23:36.842890739Z",
      "endTime": "2026-10-01T23:42:37.244899482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: Creating: virtualmachine creating or updating\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr"
Unexpected error:
    <*fmt.wrapError | 0xc00081f300>: 
    failed to create NodePool np-8l66wr: failed waiting for nodepool="np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr" in resourcegroup="rg-candidate-4-20-66r4zk7wm6rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404",
      "name": "1513980f-1a90-4c5b-865f-c0a3b6601404",
      "status": "Failed",
      "startTime": "2026-10-01T03:20:14.931117191Z",
      "endTime": "2026-10-01T03:39:22.388104494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr"
Unexpected error:
    <*fmt.wrapError | 0xc00081f300>: 
    failed to create NodePool np-8l66wr: failed waiting for nodepool="np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr" in resourcegroup="rg-candidate-4-20-66r4zk7wm6rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404",
      "name": "1513980f-1a90-4c5b-865f-c0a3b6601404",
      "status": "Failed",
      "startTime": "2026-10-01T03:20:14.931117191Z",
      "endTime": "2026-10-01T03:39:22.388104494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2105784694955053056
n/a2105477920788058112
n/a2105450319994425344
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value canno...
full failure pattern:
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 10INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-kbwzzs" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00104fe60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb6v87bszxhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-kbwzzs
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-kbwzzs" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00104fe60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb6v87bszxhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-kbwzzs
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-hwc6dp" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00164d620>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-txlxhwglc5dq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-hwc6dp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-hwc6dp" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00164d620>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-txlxhwglc5dq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-hwc6dp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-sq7dhp" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0015c4e70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-vjpbzl2jbr62/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-sq7dhp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-sq7dhp" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0015c4e70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-vjpbzl2jbr62/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-sq7dhp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (10)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105477920788058112
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 8PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009252c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-tfpfzw4vzd4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604",
      "name": "8e650b8c-c903-48db-ae29-20e1c0f74604",
      "status": "Failed",
      "startTime": "2026-10-01T23:01:41.16595227Z",
      "endTime": "2026-10-01T23:20:45.151714689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: Creating: virtualmachine creating or updating\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009252c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-tfpfzw4vzd4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604",
      "name": "8e650b8c-c903-48db-ae29-20e1c0f74604",
      "status": "Failed",
      "startTime": "2026-10-01T23:01:41.16595227Z",
      "endTime": "2026-10-01T23:20:45.151714689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: Creating: virtualmachine creating or updating\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f200c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4rdzlcbp8jqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931",
      "name": "7941e9fc-eabb-4912-b47f-1619c832d931",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:15.574287096Z",
      "endTime": "2026-10-01T23:21:15.149755975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f200c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4rdzlcbp8jqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931",
      "name": "7941e9fc-eabb-4912-b47f-1619c832d931",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:15.574287096Z",
      "endTime": "2026-10-01T23:21:15.149755975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd"
Unexpected error:
    <*fmt.wrapError | 0xc001012e40>: 
    failed to create NodePool np-6g6tzd: failed waiting for nodepool="np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd" in resourcegroup="rg-candidate-4-22-xz7x2977wmz2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "name": "3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:22.461339415Z",
      "endTime": "2026-10-01T23:41:27.247269633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: Creating: virtualmachine creating or updating\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd"
Unexpected error:
    <*fmt.wrapError | 0xc001012e40>: 
    failed to create NodePool np-6g6tzd: failed waiting for nodepool="np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd" in resourcegroup="rg-candidate-4-22-xz7x2977wmz2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "name": "3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:22.461339415Z",
      "endTime": "2026-10-01T23:41:27.247269633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: Creating: virtualmachine creating or updating\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (8)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105784694955053056
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1INT
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001c16198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-mrv84pkzs8gp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "name": "c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:38.499943334Z",
      "endTime": "2026-10-01T23:21:45.00235976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001c16198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-mrv84pkzs8gp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "name": "c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:38.499943334Z",
      "endTime": "2026-10-01T23:21:45.00235976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105784694955053056
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019901c8>: 
    resource group "oidc-wi-nq9xg56z8ptc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nq9xg56z8ptc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "name": "b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "status": "Failed",
      "startTime": "2026-10-02T00:11:10.101784102Z",
      "endTime": "2026-10-02T00:35:13.452817702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t6a9r7piu3mrhnemk9adlcvlla5mb7v still exists (deletion dispatched at 2026-10-02T00:11:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019901c8>: 
    resource group "oidc-wi-nq9xg56z8ptc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nq9xg56z8ptc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "name": "b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "status": "Failed",
      "startTime": "2026-10-02T00:11:10.101784102Z",
      "endTime": "2026-10-02T00:35:13.452817702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t6a9r7piu3mrhnemk9adlcvlla5mb7v still exists (deletion dispatched at 2026-10-02T00:11:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105784694955053056
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 0 · Oct 1: 1DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d82400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-g79pk8vm4jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "name": "d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "status": "Failed",
      "startTime": "2026-10-01T23:53:42.212331844Z",
      "endTime": "2026-10-02T00:12:43.153935456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: Creating: virtualmachine creating or updating\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d82400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-g79pk8vm4jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "name": "d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "status": "Failed",
      "startTime": "2026-10-01T23:53:42.212331844Z",
      "endTime": "2026-10-02T00:12:43.153935456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: Creating: virtualmachine creating or updating\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105784694955053056