Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 146 | 43.84% (off track) | -51.16pp | +3.08pp | 92.59% (125/135) | +0.79pp | 51.20% (64/125) | +4.80pp |
| INT | 95% - All E2E job runs | 25 | 68.00% (off track) | -27.00pp | +54.21pp | n/a | n/a | 100.00% (17/17) | +0.00pp |
| STG | 95% - All E2E job runs | 22 | 36.36% (off track) | -58.64pp | +14.14pp | n/a | n/a | 66.67% (8/12) | +41.67pp |
| PROD | 95% - All E2E job runs | 69 | 66.67% (off track) | -28.33pp | +14.58pp | n/a | n/a | 74.19% (46/62) | +19.25pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 14 | 4.01%14 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 26: 4 · Oct 27: 5 · Oct 28: 3 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 32 time(s) (16 unknown, 16 known) Firing 1: State: Resolved Started: 2026-10-29T12:26:48Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenf78afom0k6p4lvje5du442ter0n-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenf78afom0k6p4lvje5du442ter0n-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenclkh9gao5jck8ankgnd2k3u0l48-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenclkh9gao5jck8ankgnd2k3u0l48-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toencac7e983vjfhpajh2fddlt8ep68-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toencac7e983vjfhpajh2fddlt8ep68-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toen7utmmi2ofuuljqf251dkkit43s9-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toen7utmmi2ofuuljqf251dkkit43s9-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toendtu6gucjmthvacnktsi2oe747v8-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toendtu6gucjmthvacnktsi2oe747v8-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenlcpqjg4ttof16v7o0j0sosjqq95-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenlcpqjg4ttof16v7o0j0sosjqq95-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toencet327ovcj6ap2fkouscregf7c2-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toencet327ovcj6ap2fkouscregf7c2-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenke97a5bk2vegeu3khehqbm53ne6-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenke97a5bk2vegeu3khehqbm53ne6-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenm2s7n7a70o3vruq4j7savjihvb7-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenm2s7n7a70o3vruq4j7savjihvb7-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toen87v09l7pnkl9cqmk28dm4on0jv3-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toen87v09l7pnkl9cqmk28dm4on0jv3-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenkvmrf9kitsfupproge5rtqsnvpa-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenkvmrf9kitsfupproge5rtqsnvpa-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-29T12:29:47Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toend4pu9e5qqhf6cqlu1frlf44rn8h-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toend4pu9e5qqhf6cqlu1frlf44rn8h-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-29T12:29:47Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toen7oq5543lvqs2e5vlbf4bq3nnoo6-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toen7oq5543lvqs2e5vlbf4bq3nnoo6-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-29T12:29:47Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenjrtkbuag0a4j0ssqka3op79ii2u-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenjrtkbuag0a4j0ssqka3op79ii2u-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-29T12:31:49Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenk7382b2cqe5dqh3nj9qm0fraqou-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenk7382b2cqe5dqh3nj9qm0fraqou-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-29T12:31:49Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toend4jtsh1598nb94m913f43t5kvv5-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toend4jtsh1598nb94m913f43t5kvv5-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. alert fired 19 time(s) (1 unknown, 18 known) Firing 1: State: Resolved Started: 2026-10-29T11:48:42Z Ended: 2026-10-29T11:54:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6619776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2todvq5hioovckui4ebropseqsloj9is-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2todvq5hioovckui4ebropseqsloj9is-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. alert fired 44 time(s) (22 unknown, 22 known) Firing 1: State: Resolved Started: 2026-10-29T00:44:39Z Ended: 2026-10-29T00:55:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ck31s4rriq8m3aod51np4tkaadmt-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ck31s4rriq8m3aod51np4tkaadmt-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-29T00:44:39Z Ended: 2026-10-29T00:55:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4chff48f13vs6hovvitkv564933vi-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4chff48f13vs6hovvitkv564933vi-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4d8f2i0osrmij511ou0t7rfhmnlsn-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4d8f2i0osrmij511ou0t7rfhmnlsn-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cmr4pghcqg0403a5jq11dvfnlikl-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cmr4pghcqg0403a5jq11dvfnlikl-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cm85p376bf4l9af5geeqascs19re-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cm85p376bf4l9af5geeqascs19re-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cldmf4jan6oonnq2tg3nk9r52s81-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cldmf4jan6oonnq2tg3nk9r52s81-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:55:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cm3bgo3iqkg6triqkq45fn7rdlr8-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cm3bgo3iqkg6triqkq45fn7rdlr8-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-29T00:45:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4dtdrbk41153d35r2v928ecfh6bmp-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4dtdrbk41153d35r2v928ecfh6bmp-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-29T00:45:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4dgi32u5j8fk020vol0qqv1ds1k11-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4dgi32u5j8fk020vol0qqv1ds1k11-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-29T00:45:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clhtd0hi1sub7sbhcb7dhg3pba48-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clhtd0hi1sub7sbhcb7dhg3pba48-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-29T00:45:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clv3ma8s477hmmkla42d167b93ol-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clv3ma8s477hmmkla42d167b93ol-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-29T00:46:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4davicql96oofkg045duq6nghl7ff-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4davicql96oofkg045duq6nghl7ff-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-29T00:46:40Z Ended: 2026-10-29T00:55:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4e557h8kaiucvu5gvvdsgqh4o581v-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4e557h8kaiucvu5gvvdsgqh4o581v-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-29T00:46:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ct1nqdjgi22uib0cecj6up04l0m7-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ct1nqdjgi22uib0cecj6up04l0m7-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clpeml5pupvinmlvbpse0ie0hfs9-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clpeml5pupvinmlvbpse0ie0hfs9-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4darhtjunfn21raucc3dfc59v09jp-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4darhtjunfn21raucc3dfc59v09jp-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ckvnln5c26n1qo12ulvfvcdqeadr-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ckvnln5c26n1qo12ulvfvcdqeadr-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clafsmcnsm8b1l90dllklk71mg71-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clafsmcnsm8b1l90dllklk71mg71-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4d88ealur630tj4sst5o6qkjn1dgl-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4d88ealur630tj4sst5o6qkjn1dgl-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:55:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ctr3n2rtj0ed87918esf64n0iglc-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ctr3n2rtj0ed87918esf64n0iglc-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4db5tooak0klk2r251u93t1sccjm0-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4db5tooak0klk2r251u93t1sccjm0-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cl4qqobidbthts4n7r6dv1md7maq-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cl4qqobidbthts4n7r6dv1md7maq-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | alert | 14 | 4.01%14 of 349 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 26: 4 · Oct 27: 2 · Oct 28: 4 · Oct 29: 4 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T19:16:57Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.225:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-nc7qt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dbf8e658-1bc7-430e-8ce2-1f6c76f62e38" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-nc7qt (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-29T19:19:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.57:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-kml94", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8b51f9f0-c076-4060-b6fb-4a9e1396b5b6" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-kml94 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-29T19:19:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.225:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-nc7qt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dbf8e658-1bc7-430e-8ce2-1f6c76f62e38" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-nc7qt (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-29T19:19:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.57:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-kml94", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8b51f9f0-c076-4060-b6fb-4a9e1396b5b6" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-kml94 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T16:05:53Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-rdnhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="88524800-d6ff-4cb4-8768-11325673da1b" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-rdnhz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-29T16:07:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.229:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-6v8bj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cc2859a5-f39a-4c88-a623-ff48298d4334" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-6v8bj (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-29T16:07:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-rdnhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="88524800-d6ff-4cb4-8768-11325673da1b" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-rdnhz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-29T16:07:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.229:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-6v8bj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cc2859a5-f39a-4c88-a623-ff48298d4334" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-6v8bj (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T08:05:53Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-87fq6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fb1a32bd-f4c6-4a27-a3e9-8ea2cd5b63a8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-87fq6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-29T08:06:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.246:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-xfr2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0d3b843f-d25e-4240-979b-c8cbbb202083" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-xfr2d (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-29T08:08:55Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.246:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-xfr2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0d3b843f-d25e-4240-979b-c8cbbb202083" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-xfr2d (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-29T08:09:54Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-87fq6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fb1a32bd-f4c6-4a27-a3e9-8ea2cd5b63a8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-87fq6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 13 | 3.72%13 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 26: 2 · Oct 27: 3 · Oct 28: 6 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | INT, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x3674a9c4e220>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-t9dgnpvl98vn" to finish creating: GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34",
"name": "fc792b8b-e015-4d89-a0db-da7d13830c34",
"status": "Failed",
"startTime": "2026-10-29T16:16:06.194748972Z",
"endTime": "2026-10-29T16:45:10.710444768Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
correlation ID: 9f4e5eef-e698-490e-b8cf-ae403d05c79c
client request ID: 86598878-fce3-449c-9061-c59955db4018
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x3674a9c4e220>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-t9dgnpvl98vn" to finish creating: GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34",
"name": "fc792b8b-e015-4d89-a0db-da7d13830c34",
"status": "Failed",
"startTime": "2026-10-29T16:16:06.194748972Z",
"endTime": "2026-10-29T16:45:10.710444768Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
correlation ID: 9f4e5eef-e698-490e-b8cf-ae403d05c79c
client request ID: 86598878-fce3-449c-9061-c59955db4018
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x2091c74f61e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m676msdbxkvh" to finish creating: GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"name": "e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"status": "Failed",
"startTime": "2026-10-29T01:48:08.746106597Z",
"endTime": "2026-10-29T02:17:11.026048474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
correlation ID: 5c511849-df6c-49a5-a63d-d7d0d096b75d
client request ID: e632af41-8b08-40f4-bad1-f012e66c5cee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x2091c74f61e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m676msdbxkvh" to finish creating: GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"name": "e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"status": "Failed",
"startTime": "2026-10-29T01:48:08.746106597Z",
"endTime": "2026-10-29T02:17:11.026048474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
correlation ID: 5c511849-df6c-49a5-a63d-d7d0d096b75d
client request ID: e632af41-8b08-40f4-bad1-f012e66c5cee
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1fcb88002b40>:
failed to create HCP cluster np-downgrade-llw24f: failed waiting for cluster="np-downgrade-llw24f" in resourcegroup="rg-np-downgrade-llw24f-kqx7lnhpdqpq" to finish creating: GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"name": "dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"status": "Failed",
"startTime": "2026-10-28T23:17:28.994096116Z",
"endTime": "2026-10-28T23:46:34.623847458Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
correlation ID: 162093f7-f0bd-4b1e-b072-d657b497ab59
client request ID: 0bb02caa-9eaa-4069-8399-efdeced8dac8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1fcb88002b40>:
failed to create HCP cluster np-downgrade-llw24f: failed waiting for cluster="np-downgrade-llw24f" in resourcegroup="rg-np-downgrade-llw24f-kqx7lnhpdqpq" to finish creating: GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"name": "dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"status": "Failed",
"startTime": "2026-10-28T23:17:28.994096116Z",
"endTime": "2026-10-28T23:46:34.623847458Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
correlation ID: 162093f7-f0bd-4b1e-b072-d657b497ab59
client request ID: 0bb02caa-9eaa-4069-8399-efdeced8dac8
occurredContributing tests (13)
Affected runs (13) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodCrashLooping firedfull failure pattern: alert [hcp] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 11 | 3.15%11 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 26: 3 · Oct 27: 2 · Oct 28: 4 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:36:23Z Ended: 2026-10-29T17:06:22Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c0dd711d-3085-4fe4-ad56-0342b501a32c" Description: Pod ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-29T16:36:23Z Ended: 2026-10-29T17:05:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c0dd711d-3085-4fe4-ad56-0342b501a32c" Description: Pod ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:11:09Z Ended: 2026-10-29T02:38:06Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="13506377-a0a1-42df-b327-83b565596a02" Description: Pod ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-29T02:11:09Z Ended: 2026-10-29T02:39:08Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="13506377-a0a1-42df-b327-83b565596a02" Description: Pod ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:39:20Z Ended: 2026-10-28T21:05:19Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327" Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T20:40:18Z Ended: 2026-10-28T21:06:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327" Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [hcp] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 10 | 2.87%10 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 26: 2 · Oct 27: 2 · Oct 28: 4 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:49:22Z Ended: 2026-10-29T17:02:27Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-29T16:49:22Z Ended: 2026-10-29T17:02:27Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:24:07Z Ended: 2026-10-29T02:35:09Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-29T02:24:07Z Ended: 2026-10-29T02:35:09Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:52:20Z Ended: 2026-10-28T21:03:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:52:20Z Ended: 2026-10-28T21:03:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 10 | 2.87%10 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 26: 2 · Oct 27: 2 · Oct 28: 4 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:34:27Z Ended: 2026-10-29T17:02:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-29T16:45:26Z Ended: 2026-10-29T17:01:23Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:09:08Z Ended: 2026-10-29T02:34:07Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-29T02:09:08Z Ended: 2026-10-29T02:34:07Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:37:24Z Ended: 2026-10-28T21:04:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:37:24Z Ended: 2026-10-28T21:01:24Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | provision | 10 | 2.87%10 of 349 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 26: 2 · Oct 27: 4 · Oct 28: 3 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-29T04:30:23.876Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T04:30:25.398Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T04:30:26.536Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f45d1e5c6b865770f920f060f41efd0137429ae403f880e6cf3c7dda4bb749b3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1611392-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff45d1e5c6b865770f920f060f41efd0137429ae403f880e6cf3c7dda4bb749b3
time=2026-10-29T04:32:57.126Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.Resources/deployments/f45d1e5c6b865770f920f060f41efd0137429ae403f880e6cf3c7dda4bb749b3/operationStatuses/08584083586595300523\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b9d1b967-fce3-4291-995e-41596346ad76\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.Resources/deployments/cluster-fjmdbyledblew\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b9d1b967-fce3-4291-995e-41596346ad76\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j1611392-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j1611392-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-29T04:34:57.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T04:34:58.130Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T04:35:07.065Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=19b7db859db524fa83889a05d098c66f5305747330cca9423e2ad72279dea19a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1611392-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F19b7db859db524fa83889a05d098c66f5305747330cca9423e2ad72279dea19a
time=2026-10-29T04:36:37.470Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.Resources/deployments/19b7db859db524fa83889a05d098c66f5305747330cca9423e2ad72279dea19a/operationStatuses/08584083583803398033\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: df15aba7-50a9-49f1-a665-fa25966a7d0e\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-29T00:10:51.309Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T00:10:53.106Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T00:10:54.792Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4319744-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4
time=2026-10-29T00:12:55.689Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4/operationStatuses/08584083742316437961\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/cluster-6um4nhqkdznjc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j4319744-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j4319744-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-29T00:14:55.689Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T00:14:56.876Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T00:14:58.481Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4319744-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45
time=2026-10-29T00:16:28.867Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45/operationStatuses/08584083739878467732\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-28T16:43:28.511Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:43:31.273Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:43:32.448Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87
time=2026-10-28T16:45:32.811Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87/operationStatuses/08584084010736719502\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/cluster-rybldmoz73lrm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2862080-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2862080-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T16:47:32.817Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:47:34.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:47:35.068Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006
time=2026-10-28T16:49:05.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006/operationStatuses/08584084008309075646\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 9 | 2.58%9 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 26: 0 · Oct 27: 5 · Oct 28: 2 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T20:25:29Z Ended: 2026-10-29T20:31:30Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4005376-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:21:29Z Ended: 2026-10-29T02:39:31Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7406208-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T21:09:13Z Ended: 2026-10-28T21:26:12Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8906368-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 9 | 2.58%9 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 26: 0 · Oct 27: 6 · Oct 28: 1 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 3 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T12:12:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2007552-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-9889fbcdb-wfqxr", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-9889fbcdb-wfqxr has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-29T12:12:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2007552-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-9889fbcdb-6cndc", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-9889fbcdb-6cndc has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-29T12:13:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2007552-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-85d5f7ccc-bw7nz", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-85d5f7ccc-bw7nz has been in a non-ready state for longer than 5 minutes. alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6974c85f45-zzrbj", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-6974c85f45-zzrbj has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6974c85f45-p5jmb", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-6974c85f45-p5jmb has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-7cbc64f544-mxsh9", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-7cbc64f544-mxsh9 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6974c85f45-lfvfw", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-6974c85f45-lfvfw has been in a non-ready state for longer than 5 minutes. alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-64b56768b9-r7v9d", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-64b56768b9-r7v9d has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-7b9596cccb-g9thf", region="canadacentral", severity="warning" Description: Pod fleet/fleet-controller-7b9596cccb-g9thf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-t9snj", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-t9snj has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-rm24l", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-rm24l has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 7 | 2.01%7 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 26: 4 · Oct 27: 3 · Oct 28: 1 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3578474c0240>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gjfz7z4nxgmg" to finish creating: GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"name": "d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"status": "Failed",
"startTime": "2026-10-29T21:48:35.303274184Z",
"endTime": "2026-10-29T22:07:37.57729809Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-5ww8k-lpn65: NodeDoesNotExist: Node does not exist\nMachine cluster-np-1-5ww8k-56zrg: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-5ww8k-lpn65: Failed: virtualmachine failed to create or update. err: failed to get existing resource ingress-cert-le-gjfz7z4nxgmg--managed/cluster-np-1-5ww8k-lpn65 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gjfz7z4nxgmg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5ww8k-lpn65\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-thro... message truncated\nMachine cluster-np-1-5ww8k-56zrg: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
correlation ID: 443d1b16-e3d5-463c-b730-8cd4c86e6293
client request ID: 2bd9ac40-1ceb-4e60-80b5-16c2651e38ce
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3578474c0240>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gjfz7z4nxgmg" to finish creating: GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"name": "d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"status": "Failed",
"startTime": "2026-10-29T21:48:35.303274184Z",
"endTime": "2026-10-29T22:07:37.57729809Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-5ww8k-lpn65: NodeDoesNotExist: Node does not exist\nMachine cluster-np-1-5ww8k-56zrg: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-5ww8k-lpn65: Failed: virtualmachine failed to create or update. err: failed to get existing resource ingress-cert-le-gjfz7z4nxgmg--managed/cluster-np-1-5ww8k-lpn65 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gjfz7z4nxgmg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5ww8k-lpn65\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-thro... message truncated\nMachine cluster-np-1-5ww8k-56zrg: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
correlation ID: 443d1b16-e3d5-463c-b730-8cd4c86e6293
client request ID: 2bd9ac40-1ceb-4e60-80b5-16c2651e38cefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x6f3e064d560>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-nw86wtvvh485" to finish creating: GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"name": "a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"status": "Failed",
"startTime": "2026-10-29T11:24:38.055894126Z",
"endTime": "2026-10-29T11:43:42.222979982Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-fjfm9-tkhpd: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-fjfm9-tkhpd: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-nw86wtvvh485--managed/cluster-np-1-fjfm9-tkhpd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-nw86wtvvh485--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-fjfm9-tkhpd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
correlation ID: d21776c7-10dd-4392-956c-91ea278d84d3
client request ID: 0326f865-16a8-401d-88b7-c15ecd702d5e
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x6f3e064d560>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-nw86wtvvh485" to finish creating: GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"name": "a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"status": "Failed",
"startTime": "2026-10-29T11:24:38.055894126Z",
"endTime": "2026-10-29T11:43:42.222979982Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-fjfm9-tkhpd: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-fjfm9-tkhpd: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-nw86wtvvh485--managed/cluster-np-1-fjfm9-tkhpd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-nw86wtvvh485--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-fjfm9-tkhpd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
correlation ID: d21776c7-10dd-4392-956c-91ea278d84d3
client request ID: 0326f865-16a8-401d-88b7-c15ecd702d5efail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549Contributing tests (4)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 7 | 2.01%7 of 349 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 26: 0 · Oct 27: 4 · Oct 28: 1 · Oct 29: 3 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x2e578d11a500>:
DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2e578d07c700>{
msg: "DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x2e578d11a500>:
DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2e578d07c700>{
msg: "DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-kns88h"
Unexpected error:
<*fmt.wrapError | 0x35a5d6ab6c80>:
DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x35a5d645c1c0>{
msg: "DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-kns88h"
Unexpected error:
<*fmt.wrapError | 0x35a5d6ab6c80>:
DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x35a5d645c1c0>{
msg: "DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0x1942e140ed80>:
DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1942e1526600>{
msg: "DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0x1942e140ed80>:
DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1942e1526600>{
msg: "DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (8)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 5 | 1.43%5 of 349 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 26: 1 · Oct 27: 2 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1full failure pattern: VerifyAllAPIServicesAvailable failed: apiservice/v1beta1 Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 4 | 1.15%4 of 349 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 2 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
<*errors.joinError | 0x12b8c8eac018>:
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
errs: [
<*fmt.wrapError | 0x12b8c80a4040>{
msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*errors.joinError | 0x12b8c8ead218>{
errs: [
<*errors.errorString | 0x12b8c9007710>{
s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
},
],
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
<*errors.joinError | 0x12b8c8eac018>:
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
errs: [
<*fmt.wrapError | 0x12b8c80a4040>{
msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*errors.joinError | 0x12b8c8ead218>{
errs: [
<*errors.errorString | 0x12b8c9007710>{
s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.27:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
},
],
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
<*errors.joinError | 0x311de884c030>:
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
errs: [
<*fmt.wrapError | 0x311de7428000>{
msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*errors.joinError | 0x311de82372a8>{
errs: [
<*errors.errorString | 0x311de8b96230>{
s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
},
],
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
<*errors.joinError | 0x311de884c030>:
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
errs: [
<*fmt.wrapError | 0x311de7428000>{
msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*errors.joinError | 0x311de82372a8>{
errs: [
<*errors.errorString | 0x311de8b96230>{
s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.129.0.21:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
},
],
},
},
],
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
<*errors.joinError | 0xc83633402b8>:
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
errs: [
<*fmt.wrapError | 0xc83630bcee0>{
msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*errors.joinError | 0xc8362e9fb78>{
errs: [
<*errors.errorString | 0xc83633b5610>{
s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
},
],
},
},
],
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
<*errors.joinError | 0xc83633402b8>:
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
errs: [
<*fmt.wrapError | 0xc83630bcee0>{
msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*errors.joinError | 0xc8362e9fb78>{
errs: [
<*errors.errorString | 0xc83633b5610>{
s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
},
],
},
},
],
}Contributing tests (3)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
at least one schedule should have a LastBackupTimefull failure pattern: at least one schedule should have a LastBackupTime Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 23 | 6.59%23 of 349 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 26: 1 · Oct 27: 22 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 6 | 1.72%6 of 349 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 26: 0 · Oct 27: 2 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x1ddd7d2831b8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zk7rbqmsp2zd" to finish creating: GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"name": "cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"status": "Failed",
"startTime": "2026-10-29T00:42:47.491540846Z",
"endTime": "2026-10-29T01:01:51.350365806Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-ztzhx-l97nr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-ztzhx-l97nr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cilium-np-ztzhx-547rj: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-zk7rbqmsp2zd--managed/cluster-cilium-np-ztzhx-547rj-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Network/networkInterfaces/cluster-cilium-np-ztzhx-547rj-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/a... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
correlation ID: 52dc9816-9bf4-4c02-9a62-7105502ff251
client request ID: 3591f936-ad0d-47b0-b46a-a9f82bf4e6ae
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x1ddd7d2831b8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zk7rbqmsp2zd" to finish creating: GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"name": "cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"status": "Failed",
"startTime": "2026-10-29T00:42:47.491540846Z",
"endTime": "2026-10-29T01:01:51.350365806Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-ztzhx-l97nr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-ztzhx-l97nr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cilium-np-ztzhx-547rj: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-zk7rbqmsp2zd--managed/cluster-cilium-np-ztzhx-547rj-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Network/networkInterfaces/cluster-cilium-np-ztzhx-547rj-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/a... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
correlation ID: 52dc9816-9bf4-4c02-9a62-7105502ff251
client request ID: 3591f936-ad0d-47b0-b46a-a9f82bf4e6ae
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurredContributing tests (3)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 5 | 1.43%5 of 349 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 5 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-8pc9jb"
Unexpected error:
<*errors.joinError | 0x19fbe63ec288>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-8pc9jb-xk8t4-5mzft": nodes "cluster-np-8pc9jb-xk8t4-5mzft" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x19fbe659e180>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e160>{
msg: "failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e140>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x19fbe658ea00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-8pc9jb-xk8t4-5mzft",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-8pc9jb"
Unexpected error:
<*errors.joinError | 0x19fbe63ec288>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-8pc9jb-xk8t4-5mzft": nodes "cluster-np-8pc9jb-xk8t4-5mzft" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x19fbe659e180>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e160>{
msg: "failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e140>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x19fbe658ea00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-8pc9jb-xk8t4-5mzft",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-spgbv9"
Unexpected error:
<*errors.joinError | 0xb85fe22660>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-spgbv9-bvtl9-fgbn6": nodes "cluster-np-spgbv9-bvtl9-fgbn6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xb8604081a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408180>{
msg: "failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408160>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xb860403860>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-spgbv9-bvtl9-fgbn6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-spgbv9"
Unexpected error:
<*errors.joinError | 0xb85fe22660>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-spgbv9-bvtl9-fgbn6": nodes "cluster-np-spgbv9-bvtl9-fgbn6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xb8604081a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408180>{
msg: "failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408160>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xb860403860>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-spgbv9-bvtl9-fgbn6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-j2tgkf"
Unexpected error:
<*errors.joinError | 0xf595dece090>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-j2tgkf-q8c9h-j47g6": nodes "cluster-np-j2tgkf-q8c9h-j47g6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xf595db0f4c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f4a0>{
msg: "failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f480>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xf595e31e6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-j2tgkf-q8c9h-j47g6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-j2tgkf"
Unexpected error:
<*errors.joinError | 0xf595dece090>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-j2tgkf-q8c9h-j47g6": nodes "cluster-np-j2tgkf-q8c9h-j47g6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xf595db0f4c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f4a0>{
msg: "failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f480>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xf595e31e6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-j2tgkf-q8c9h-j47g6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] hypershiftOperatorPodsDown firedfull failure pattern: alert [svc] hypershiftOperatorPodsDown fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | alert | 5 | 1.43%5 of 349 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 4 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:11:35Z Ended: 2026-10-29T16:17:37Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j8756352-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j8756352-mgmt-2 alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T14:55:08Z Ended: 2026-10-29T15:01:06Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j0084992-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j0084992-mgmt-2 alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T12:14:15Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j2007552-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j2007552-mgmt-2 Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | unknown | 3 | 12.00%3 of 25 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV, PROD, STG | ||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sl7v6jcfzpzn/cluster-candidate-5-0-rw67x7 should provision
Unexpected error:
<*fmt.wrapError | 0x320a55350300>:
failed to create HCP cluster cluster-candidate-5-0-rw67x7: failed waiting for cluster="cluster-candidate-5-0-rw67x7" in resourcegroup="rg-candidate-5-0-sl7v6jcfzpzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"name": "e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"status": "Failed",
"startTime": "2026-10-29T01:24:15.287612604Z",
"endTime": "2026-10-29T01:53:18.05722195Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
correlation ID: c2102baf-9a88-4da3-a6f8-854193b8aaa9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sl7v6jcfzpzn/cluster-candidate-5-0-rw67x7 should provision
Unexpected error:
<*fmt.wrapError | 0x320a55350300>:
failed to create HCP cluster cluster-candidate-5-0-rw67x7: failed waiting for cluster="cluster-candidate-5-0-rw67x7" in resourcegroup="rg-candidate-5-0-sl7v6jcfzpzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"name": "e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"status": "Failed",
"startTime": "2026-10-29T01:24:15.287612604Z",
"endTime": "2026-10-29T01:53:18.05722195Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
correlation ID: c2102baf-9a88-4da3-a6f8-854193b8aaa9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurredContributing tests (3)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.00%1 of 25 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.00%1 of 25 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.00%1 of 25 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-10-29T19:59:31Z"}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.00%1 of 25 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.00%1 of 25 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x866rm"
Unexpected error:
<*errors.joinError | 0x2cfb1b0a33e0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-x866rm-lmp2b-9wggz": nodes "cluster-np-x866rm-lmp2b-9wggz" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2cfb1c1b05e0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05c0>{
msg: "failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05a0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2cfb1c1b6460>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-x866rm-lmp2b-9wggz",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x866rm"
Unexpected error:
<*errors.joinError | 0x2cfb1b0a33e0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-x866rm-lmp2b-9wggz": nodes "cluster-np-x866rm-lmp2b-9wggz" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2cfb1c1b05e0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05c0>{
msg: "failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05a0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2cfb1c1b6460>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-x866rm-lmp2b-9wggz",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 54.55% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 54.55% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 63.64% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 72.73% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 72.73% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 9.09%2 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsOct 26: 2 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (5)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 9.09%2 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 26: 2 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 10 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-vrx699" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xb37c723e620>:
failed to create HCP cluster cluster-zstream-4-22-vrx699: failed waiting for cluster="cluster-zstream-4-22-vrx699" in resourcegroup="rg-zstream-upgrade-4-22-mk5975tqzhdb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa",
"name": "f522dee9-4911-4a39-a8b7-66d53ae810aa",
"status": "Failed",
"startTime": "2026-10-28T21:21:44.905508951Z",
"endTime": "2026-10-28T21:50:49.351811692Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
correlation ID: fa068c44-117e-4214-8c6d-206698ccb0ad
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-vrx699" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xb37c723e620>:
failed to create HCP cluster cluster-zstream-4-22-vrx699: failed waiting for cluster="cluster-zstream-4-22-vrx699" in resourcegroup="rg-zstream-upgrade-4-22-mk5975tqzhdb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa",
"name": "f522dee9-4911-4a39-a8b7-66d53ae810aa",
"status": "Failed",
"startTime": "2026-10-28T21:21:44.905508951Z",
"endTime": "2026-10-28T21:50:49.351811692Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
correlation ID: fa068c44-117e-4214-8c6d-206698ccb0ad
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not existfull failure pattern: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-<id>])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-<id>]) Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:886]: Timed out after 900.004s.
node pool np-1 never detached the day 2 MI /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/acr-pull-2pg5nqg2c8wm/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi-2 after removal
Expected success, but got an error:
<*errors.errorString | 0x18a468a961b0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:886]: Timed out after 900.004s.
node pool np-1 never detached the day 2 MI /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/acr-pull-2pg5nqg2c8wm/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi-2 after removal
Expected success, but got an error:
<*errors.errorString | 0x18a468a961b0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.55%1 of 22 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 50.00% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 50.00% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 58.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 58.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 58.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 7 | 10.14%7 of 69 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 26: 0 · Oct 27: 2 · Oct 28: 19 · Oct 29: 3 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV, INT, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x1def056ca980>:
failed to create HCP cluster ea-listzz8ctk: failed waiting for cluster="ea-listzz8ctk" in resourcegroup="ea-list-4jljqltdpf22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51",
"name": "671f379d-8472-4150-82a8-d71efe3bef51",
"status": "Failed",
"startTime": "2026-10-29T22:40:05.142063562Z",
"endTime": "2026-10-29T23:09:09.015450129Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
correlation ID: 462ab43f-575c-42b1-ab0d-20c41a4d0046
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x1def056ca980>:
failed to create HCP cluster ea-listzz8ctk: failed waiting for cluster="ea-listzz8ctk" in resourcegroup="ea-list-4jljqltdpf22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51",
"name": "671f379d-8472-4150-82a8-d71efe3bef51",
"status": "Failed",
"startTime": "2026-10-29T22:40:05.142063562Z",
"endTime": "2026-10-29T23:09:09.015450129Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
correlation ID: 462ab43f-575c-42b1-ab0d-20c41a4d0046
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x241c81f32b80>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-58txd54zlccz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"name": "c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"status": "Failed",
"startTime": "2026-10-29T14:34:25.586865421Z",
"endTime": "2026-10-29T15:03:34.559330088Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
correlation ID: 5c469ba9-6c92-4dd6-9ba5-04a68fee9ef5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x241c81f32b80>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-58txd54zlccz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"name": "c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"status": "Failed",
"startTime": "2026-10-29T14:34:25.586865421Z",
"endTime": "2026-10-29T15:03:34.559330088Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
correlation ID: 5c469ba9-6c92-4dd6-9ba5-04a68fee9ef5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fw4jcb" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x5b710c1e4e0>:
failed to create HCP cluster cp-ystream-upgrade-4-23-fw4jcb: failed waiting for cluster="cp-ystream-upgrade-4-23-fw4jcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-sftgrsslrwdw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c",
"name": "b53af301-d50b-4096-b43d-a7144a6a296c",
"status": "Failed",
"startTime": "2026-10-29T06:25:50.628347504Z",
"endTime": "2026-10-29T06:54:52.041701794Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
correlation ID: 4b2677ce-b17f-435d-9634-3583d8f686b3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fw4jcb" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x5b710c1e4e0>:
failed to create HCP cluster cp-ystream-upgrade-4-23-fw4jcb: failed waiting for cluster="cp-ystream-upgrade-4-23-fw4jcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-sftgrsslrwdw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c",
"name": "b53af301-d50b-4096-b43d-a7144a6a296c",
"status": "Failed",
"startTime": "2026-10-29T06:25:50.628347504Z",
"endTime": "2026-10-29T06:54:52.041701794Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
correlation ID: 4b2677ce-b17f-435d-9634-3583d8f686b3
client request ID: unavailable
occurredContributing tests (17)
Affected runs (7)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment | e2e | 1 | 1.45%1 of 69 job runs affected | NoiseSignal: Noise — struct/object fragment | Show trend detailsOct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 4 | 5.80%4 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 2 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2535834a99c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rbmj59bntg5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1",
"name": "49137998-c82d-463c-9cd4-a8775024ebc1",
"status": "Failed",
"startTime": "2026-10-29T04:26:18.62948754Z",
"endTime": "2026-10-29T04:45:22.348417217Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
correlation ID: 40719e78-c8c1-4a49-a0a8-7e5f3f0cfa60
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2535834a99c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rbmj59bntg5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1",
"name": "49137998-c82d-463c-9cd4-a8775024ebc1",
"status": "Failed",
"startTime": "2026-10-29T04:26:18.62948754Z",
"endTime": "2026-10-29T04:45:22.348417217Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
correlation ID: 40719e78-c8c1-4a49-a0a8-7e5f3f0cfa60
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0xa04a5fdf140>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ffbhgrn8qbrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747",
"name": "ade1a10f-df33-4040-9e11-b9632ef21747",
"status": "Failed",
"startTime": "2026-10-29T01:55:59.309285263Z",
"endTime": "2026-10-29T02:15:01.505600334Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
correlation ID: 0ca07d4a-0c93-4a4d-806f-591a9bde4d58
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0xa04a5fdf140>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ffbhgrn8qbrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747",
"name": "ade1a10f-df33-4040-9e11-b9632ef21747",
"status": "Failed",
"startTime": "2026-10-29T01:55:59.309285263Z",
"endTime": "2026-10-29T02:15:01.505600334Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
correlation ID: 0ca07d4a-0c93-4a4d-806f-591a9bde4d58
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 16 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x22868c3a2660>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"status": "Failed",
"startTime": "2026-10-28T19:24:09.588021132Z",
"endTime": "2026-10-28T19:53:13.758028769Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x22868c3a2660>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"status": "Failed",
"startTime": "2026-10-28T19:24:09.588021132Z",
"endTime": "2026-10-28T19:53:13.758028769Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x263f7a1ecb40>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
"name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
"status": "Failed",
"startTime": "2026-10-28T20:14:34.589492538Z",
"endTime": "2026-10-28T20:43:34.88292258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x263f7a1ecb40>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
"name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
"status": "Failed",
"startTime": "2026-10-28T20:14:34.589492538Z",
"endTime": "2026-10-28T20:43:34.88292258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
<*fmt.wrapError | 0x2d29037be7e0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
"name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
"status": "Failed",
"startTime": "2026-10-28T20:12:38.563887566Z",
"endTime": "2026-10-28T20:41:44.890635353Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
<*fmt.wrapError | 0x2d29037be7e0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
"name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
"status": "Failed",
"startTime": "2026-10-28T20:12:38.563887566Z",
"endTime": "2026-10-28T20:41:44.890635353Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
client request ID: unavailable
occurredContributing tests (13)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 14 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
<*fmt.wrapError | 0x15a8256fc4a0>:
failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"status": "Failed",
"startTime": "2026-10-28T20:04:15.708912895Z",
"endTime": "2026-10-28T20:33:16.216619801Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
<*fmt.wrapError | 0x15a8256fc4a0>:
failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"status": "Failed",
"startTime": "2026-10-28T20:04:15.708912895Z",
"endTime": "2026-10-28T20:33:16.216619801Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xd149dec0ae0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.936404311Z",
"endTime": "2026-10-28T19:53:23.678300779Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xd149dec0ae0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.936404311Z",
"endTime": "2026-10-28T19:53:23.678300779Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x3b3d00e688d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"status": "Failed",
"startTime": "2026-10-28T20:59:31.361013478Z",
"endTime": "2026-10-28T21:28:40.621642045Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x3b3d00e688d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"status": "Failed",
"startTime": "2026-10-28T20:59:31.361013478Z",
"endTime": "2026-10-28T21:28:40.621642045Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
client request ID: unavailable
occurredContributing tests (11)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 12 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
<*fmt.wrapError | 0xba6b52aa80>:
failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"status": "Failed",
"startTime": "2026-10-28T20:10:20.02531103Z",
"endTime": "2026-10-28T20:39:25.022579343Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
correlation ID: 15324628-570b-42b9-80a7-dbd435939211
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
<*fmt.wrapError | 0xba6b52aa80>:
failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"status": "Failed",
"startTime": "2026-10-28T20:10:20.02531103Z",
"endTime": "2026-10-28T20:39:25.022579343Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
correlation ID: 15324628-570b-42b9-80a7-dbd435939211
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
<*fmt.wrapError | 0x104ef3fa7e0>:
failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
"name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
"status": "Failed",
"startTime": "2026-10-28T20:10:48.049950576Z",
"endTime": "2026-10-28T20:39:55.034999436Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
<*fmt.wrapError | 0x104ef3fa7e0>:
failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
"name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
"status": "Failed",
"startTime": "2026-10-28T20:10:48.049950576Z",
"endTime": "2026-10-28T20:39:55.034999436Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x1da2d78427a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"status": "Failed",
"startTime": "2026-10-28T19:26:05.081696809Z",
"endTime": "2026-10-28T19:55:13.864538233Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x1da2d78427a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"status": "Failed",
"startTime": "2026-10-28T19:26:05.081696809Z",
"endTime": "2026-10-28T19:55:13.864538233Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
client request ID: unavailable
occurredContributing tests (9)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsOct 26: 5 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (8)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x34cbd3d68900>:
failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
"name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
"status": "Failed",
"startTime": "2026-10-28T20:14:32.731497539Z",
"endTime": "2026-10-28T20:43:34.905565303Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x34cbd3d68900>:
failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
"name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
"status": "Failed",
"startTime": "2026-10-28T20:14:32.731497539Z",
"endTime": "2026-10-28T20:43:34.905565303Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x118c82e68960>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"status": "Failed",
"startTime": "2026-10-28T20:58:03.268943167Z",
"endTime": "2026-10-28T21:27:10.630246232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x118c82e68960>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"status": "Failed",
"startTime": "2026-10-28T20:58:03.268943167Z",
"endTime": "2026-10-28T21:27:10.630246232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
<*fmt.wrapError | 0x3e94b832e820>:
failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
"name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
"status": "Failed",
"startTime": "2026-10-28T20:06:06.856819198Z",
"endTime": "2026-10-28T20:35:15.000751357Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
<*fmt.wrapError | 0x3e94b832e820>:
failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
"name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
"status": "Failed",
"startTime": "2026-10-28T20:06:06.856819198Z",
"endTime": "2026-10-28T20:35:15.000751357Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x1d67385a4720>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"status": "Failed",
"startTime": "2026-10-28T19:24:18.787179838Z",
"endTime": "2026-10-28T19:53:23.770490111Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x1d67385a4720>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"status": "Failed",
"startTime": "2026-10-28T19:24:18.787179838Z",
"endTime": "2026-10-28T19:53:23.770490111Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x23a60cd6e680>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"status": "Failed",
"startTime": "2026-10-28T19:24:25.094565507Z",
"endTime": "2026-10-28T19:53:33.692195544Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x23a60cd6e680>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"status": "Failed",
"startTime": "2026-10-28T19:24:25.094565507Z",
"endTime": "2026-10-28T19:53:33.692195544Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x34142edd6a40>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
"name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
"status": "Failed",
"startTime": "2026-10-28T19:24:14.723008935Z",
"endTime": "2026-10-28T19:53:23.809094912Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x34142edd6a40>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
"name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
"status": "Failed",
"startTime": "2026-10-28T19:24:14.723008935Z",
"endTime": "2026-10-28T19:53:23.809094912Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x2a9a9c302780>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"status": "Failed",
"startTime": "2026-10-28T20:14:46.976794807Z",
"endTime": "2026-10-28T20:43:55.053369978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x2a9a9c302780>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"status": "Failed",
"startTime": "2026-10-28T20:14:46.976794807Z",
"endTime": "2026-10-28T20:43:55.053369978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0xe23c97da540>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"status": "Failed",
"startTime": "2026-10-28T20:28:51.316103778Z",
"endTime": "2026-10-28T20:57:55.004654509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0xe23c97da540>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"status": "Failed",
"startTime": "2026-10-28T20:28:51.316103778Z",
"endTime": "2026-10-28T20:57:55.004654509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x38a796b1a740>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
"name": "499291aa-b279-429d-80ba-7786fc6cfa93",
"status": "Failed",
"startTime": "2026-10-28T20:38:17.873619905Z",
"endTime": "2026-10-28T21:07:24.99192406Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x38a796b1a740>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
"name": "499291aa-b279-429d-80ba-7786fc6cfa93",
"status": "Failed",
"startTime": "2026-10-28T20:38:17.873619905Z",
"endTime": "2026-10-28T21:07:24.99192406Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1c7b6c0a8440>:
failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
"name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
"status": "Failed",
"startTime": "2026-10-28T20:56:27.61784978Z",
"endTime": "2026-10-28T21:25:30.606619249Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1c7b6c0a8440>:
failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
"name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
"status": "Failed",
"startTime": "2026-10-28T20:56:27.61784978Z",
"endTime": "2026-10-28T21:25:30.606619249Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 26: 4 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (5)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0x2c03b21d6c20>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"status": "Failed",
"startTime": "2026-10-28T20:14:37.658468995Z",
"endTime": "2026-10-28T20:43:44.89685741Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0x2c03b21d6c20>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"status": "Failed",
"startTime": "2026-10-28T20:14:37.658468995Z",
"endTime": "2026-10-28T20:43:44.89685741Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurredContributing tests (4)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x14a5e2008300>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
"name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
"status": "Failed",
"startTime": "2026-10-28T19:26:12.743400806Z",
"endTime": "2026-10-28T19:55:13.7057976Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x14a5e2008300>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
"name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
"status": "Failed",
"startTime": "2026-10-28T19:26:12.743400806Z",
"endTime": "2026-10-28T19:55:13.7057976Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x1683f5097100>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"status": "Failed",
"startTime": "2026-10-28T19:28:18.023653922Z",
"endTime": "2026-10-28T19:57:23.841118908Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x1683f5097100>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"status": "Failed",
"startTime": "2026-10-28T19:28:18.023653922Z",
"endTime": "2026-10-28T19:57:23.841118908Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x13016dc5e5e0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.851456324Z",
"endTime": "2026-10-28T20:41:34.873934441Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x13016dc5e5e0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.851456324Z",
"endTime": "2026-10-28T20:41:34.873934441Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xfa629f24ea0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xfa629c70d20>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xfa629f24ea0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xfa629c70d20>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed Contributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 26: 1 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.90%2 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 1 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x243ce2483b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-r4jthp6b8c2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67",
"name": "4f6eb8f6-434d-4041-9e38-be33ab685c67",
"status": "Failed",
"startTime": "2026-10-29T13:48:20.576616469Z",
"endTime": "2026-10-29T14:07:20.745392823Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
correlation ID: 72c0068c-b7b6-49ea-9082-ad530d20a571
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x243ce2483b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-r4jthp6b8c2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67",
"name": "4f6eb8f6-434d-4041-9e38-be33ab685c67",
"status": "Failed",
"startTime": "2026-10-29T13:48:20.576616469Z",
"endTime": "2026-10-29T14:07:20.745392823Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
correlation ID: 72c0068c-b7b6-49ea-9082-ad530d20a571
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x26d436e0e5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rgpk9lmtvnvv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105",
"name": "357cbb72-4fad-4dd1-860d-138b89ef8105",
"status": "Failed",
"startTime": "2026-10-28T23:13:44.545799939Z",
"endTime": "2026-10-28T23:32:49.563451274Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pkf64-8lv9b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pkf64-8lv9b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
correlation ID: 8689aa1d-b711-4817-af27-601b622fb10c
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x26d436e0e5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rgpk9lmtvnvv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105",
"name": "357cbb72-4fad-4dd1-860d-138b89ef8105",
"status": "Failed",
"startTime": "2026-10-28T23:13:44.545799939Z",
"endTime": "2026-10-28T23:32:49.563451274Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pkf64-8lv9b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pkf64-8lv9b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
correlation ID: 8689aa1d-b711-4817-af27-601b622fb10c
client request ID: unavailableContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0xc983e72b20>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
"name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.13733405Z",
"endTime": "2026-10-28T19:53:23.880336461Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0xc983e72b20>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
"name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.13733405Z",
"endTime": "2026-10-28T19:53:23.880336461Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3981bcd4a5c0>:
failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
"name": "fa758444-9da4-4261-b65a-fb9af0416aae",
"status": "Failed",
"startTime": "2026-10-28T20:11:05.048147318Z",
"endTime": "2026-10-28T20:40:15.036753709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3981bcd4a5c0>:
failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
"name": "fa758444-9da4-4261-b65a-fb9af0416aae",
"status": "Failed",
"startTime": "2026-10-28T20:11:05.048147318Z",
"endTime": "2026-10-28T20:40:15.036753709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x9790da22600>:
failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
"name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
"status": "Failed",
"startTime": "2026-10-28T20:55:25.009646036Z",
"endTime": "2026-10-28T21:24:30.625209621Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x9790da22600>:
failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
"name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
"status": "Failed",
"startTime": "2026-10-28T20:55:25.009646036Z",
"endTime": "2026-10-28T21:24:30.625209621Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
client request ID: unavailable
occurredContributing tests (6)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x16967f5def20>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
"name": "54478daa-47e5-4530-91c2-342cc29a1749",
"status": "Failed",
"startTime": "2026-10-28T19:24:21.628677591Z",
"endTime": "2026-10-28T19:53:23.838328082Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x16967f5def20>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
"name": "54478daa-47e5-4530-91c2-342cc29a1749",
"status": "Failed",
"startTime": "2026-10-28T19:24:21.628677591Z",
"endTime": "2026-10-28T19:53:23.838328082Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d00e53ba4e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"status": "Failed",
"startTime": "2026-10-28T19:24:17.131948207Z",
"endTime": "2026-10-28T19:53:23.815769706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d00e53ba4e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"status": "Failed",
"startTime": "2026-10-28T19:24:17.131948207Z",
"endTime": "2026-10-28T19:53:23.815769706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x1c6ab1a4a4c0>:
failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
"name": "13f26b26-2324-48a4-82df-d8b324b13996",
"status": "Failed",
"startTime": "2026-10-28T20:52:21.783931077Z",
"endTime": "2026-10-28T21:21:30.723052246Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x1c6ab1a4a4c0>:
failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
"name": "13f26b26-2324-48a4-82df-d8b324b13996",
"status": "Failed",
"startTime": "2026-10-28T20:52:21.783931077Z",
"endTime": "2026-10-28T21:21:30.723052246Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsOct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x2df91d94ebc0>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
"name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
"status": "Failed",
"startTime": "2026-10-28T19:52:14.429975606Z",
"endTime": "2026-10-28T20:21:16.149275515Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x2df91d94ebc0>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
"name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
"status": "Failed",
"startTime": "2026-10-28T19:52:14.429975606Z",
"endTime": "2026-10-28T20:21:16.149275515Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x219ad233aca0>:
failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
"name": "e768a2f1-bf75-4afe-9acf-741914b73091",
"status": "Failed",
"startTime": "2026-10-28T20:11:31.970768227Z",
"endTime": "2026-10-28T20:40:35.054714998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x219ad233aca0>:
failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
"name": "e768a2f1-bf75-4afe-9acf-741914b73091",
"status": "Failed",
"startTime": "2026-10-28T20:11:31.970768227Z",
"endTime": "2026-10-28T20:40:35.054714998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, ignition-server-proxy, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1c9f11d305c0>:
failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
"name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
"status": "Failed",
"startTime": "2026-10-28T20:57:10.002650956Z",
"endTime": "2026-10-28T21:26:10.63929862Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1c9f11d305c0>:
failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
"name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
"status": "Failed",
"startTime": "2026-10-28T20:57:10.002650956Z",
"endTime": "2026-10-28T21:26:10.63929862Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-controller-manager, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x3da3f19209c0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"status": "Failed",
"startTime": "2026-10-28T20:14:36.875741431Z",
"endTime": "2026-10-28T20:43:44.887394978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
correlation ID: 84abeaa9-4051-418b-b066-32246401955a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x3da3f19209c0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"status": "Failed",
"startTime": "2026-10-28T20:14:36.875741431Z",
"endTime": "2026-10-28T20:43:44.887394978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
correlation ID: 84abeaa9-4051-418b-b066-32246401955a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x2840a3490800>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
"name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
"status": "Failed",
"startTime": "2026-10-28T20:48:29.562853635Z",
"endTime": "2026-10-28T21:17:30.601011159Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x2840a3490800>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
"name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
"status": "Failed",
"startTime": "2026-10-28T20:48:29.562853635Z",
"endTime": "2026-10-28T21:17:30.601011159Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x3290e061a680>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"status": "Failed",
"startTime": "2026-10-28T20:57:27.436526224Z",
"endTime": "2026-10-28T21:26:30.622804028Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x3290e061a680>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"status": "Failed",
"startTime": "2026-10-28T20:57:27.436526224Z",
"endTime": "2026-10-28T21:26:30.622804028Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x1d947782ea20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
"name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
"status": "Failed",
"startTime": "2026-10-28T20:01:08.393167298Z",
"endTime": "2026-10-28T20:30:16.146752142Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x1d947782ea20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
"name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
"status": "Failed",
"startTime": "2026-10-28T20:01:08.393167298Z",
"endTime": "2026-10-28T20:30:16.146752142Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x28ad82ebac20>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
"name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
"status": "Failed",
"startTime": "2026-10-28T20:01:11.67931549Z",
"endTime": "2026-10-28T20:30:16.178438369Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x28ad82ebac20>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
"name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
"status": "Failed",
"startTime": "2026-10-28T20:01:11.67931549Z",
"endTime": "2026-10-28T20:30:16.178438369Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, kube-controller-manager, kube-scheduler]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x1308b97a65a0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"status": "Failed",
"startTime": "2026-10-28T20:58:09.127282806Z",
"endTime": "2026-10-28T21:27:10.682711777Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
correlation ID: 49b88924-064f-4f52-9480-109cf607a390
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x1308b97a65a0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"status": "Failed",
"startTime": "2026-10-28T20:58:09.127282806Z",
"endTime": "2026-10-28T21:27:10.682711777Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
correlation ID: 49b88924-064f-4f52-9480-109cf607a390
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
<*fmt.wrapError | 0xd7eab0d0620>:
failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
"name": "36728157-067b-4375-a959-9c9ce43e534b",
"status": "Failed",
"startTime": "2026-10-28T20:56:26.269135944Z",
"endTime": "2026-10-28T21:25:30.687402107Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
<*fmt.wrapError | 0xd7eab0d0620>:
failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
"name": "36728157-067b-4375-a959-9c9ce43e534b",
"status": "Failed",
"startTime": "2026-10-28T20:56:26.269135944Z",
"endTime": "2026-10-28T21:25:30.687402107Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: interfaces creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.45%1 of 69 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 · Nov 2: 0 · Nov 3: 0 · Nov 4: 0 · Nov 5: 0 · Nov 6: 0 · Nov 7: 0 · Nov 8: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||