Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 169 | 43.79% (off track) | -51.21pp | -4.19pp | 92.11% (140/152) | -1.35pp | 52.86% (74/140) | -0.01pp |
| INT | 95% - All E2E job runs | 38 | 50.00% (off track) | -45.00pp | +37.50pp | n/a | n/a | 100.00% (19/19) | +0.00pp |
| STG | 95% - All E2E job runs | 28 | 39.29% (off track) | -55.71pp | +23.90pp | n/a | n/a | 61.11% (11/18) | +41.11pp |
| PROD | 95% - All E2E job runs | 77 | 64.94% (off track) | -30.06pp | +8.04pp | n/a | n/a | 70.42% (50/71) | +10.42pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 15 | 3.62%15 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 23: 3 · Oct 24: 0 · Oct 25: 1 · Oct 26: 4 · Oct 27: 5 · Oct 28: 3 · Oct 29: 2 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 32 time(s) (16 unknown, 16 known) Firing 1: State: Resolved Started: 2026-10-29T12:26:48Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenf78afom0k6p4lvje5du442ter0n-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenf78afom0k6p4lvje5du442ter0n-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenclkh9gao5jck8ankgnd2k3u0l48-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenclkh9gao5jck8ankgnd2k3u0l48-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toencac7e983vjfhpajh2fddlt8ep68-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toencac7e983vjfhpajh2fddlt8ep68-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toen7utmmi2ofuuljqf251dkkit43s9-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toen7utmmi2ofuuljqf251dkkit43s9-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-29T12:27:49Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toendtu6gucjmthvacnktsi2oe747v8-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toendtu6gucjmthvacnktsi2oe747v8-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenlcpqjg4ttof16v7o0j0sosjqq95-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenlcpqjg4ttof16v7o0j0sosjqq95-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toencet327ovcj6ap2fkouscregf7c2-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toencet327ovcj6ap2fkouscregf7c2-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenke97a5bk2vegeu3khehqbm53ne6-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenke97a5bk2vegeu3khehqbm53ne6-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenm2s7n7a70o3vruq4j7savjihvb7-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenm2s7n7a70o3vruq4j7savjihvb7-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toen87v09l7pnkl9cqmk28dm4on0jv3-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toen87v09l7pnkl9cqmk28dm4on0jv3-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-29T12:28:52Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenkvmrf9kitsfupproge5rtqsnvpa-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenkvmrf9kitsfupproge5rtqsnvpa-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-29T12:29:47Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toend4pu9e5qqhf6cqlu1frlf44rn8h-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toend4pu9e5qqhf6cqlu1frlf44rn8h-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-29T12:29:47Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toen7oq5543lvqs2e5vlbf4bq3nnoo6-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toen7oq5543lvqs2e5vlbf4bq3nnoo6-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-29T12:29:47Z Ended: 2026-10-29T12:37:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenjrtkbuag0a4j0ssqka3op79ii2u-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenjrtkbuag0a4j0ssqka3op79ii2u-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-29T12:31:49Z Ended: 2026-10-29T12:37:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toenk7382b2cqe5dqh3nj9qm0fraqou-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toenk7382b2cqe5dqh3nj9qm0fraqou-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-29T12:31:49Z Ended: 2026-10-29T12:37:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0502016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2toend4jtsh1598nb94m913f43t5kvv5-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2toend4jtsh1598nb94m913f43t5kvv5-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. alert fired 19 time(s) (1 unknown, 18 known) Firing 1: State: Resolved Started: 2026-10-29T11:48:42Z Ended: 2026-10-29T11:54:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6619776-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2todvq5hioovckui4ebropseqsloj9is-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2todvq5hioovckui4ebropseqsloj9is-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. alert fired 44 time(s) (22 unknown, 22 known) Firing 1: State: Resolved Started: 2026-10-29T00:44:39Z Ended: 2026-10-29T00:55:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ck31s4rriq8m3aod51np4tkaadmt-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ck31s4rriq8m3aod51np4tkaadmt-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-29T00:44:39Z Ended: 2026-10-29T00:55:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4chff48f13vs6hovvitkv564933vi-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4chff48f13vs6hovvitkv564933vi-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4d8f2i0osrmij511ou0t7rfhmnlsn-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4d8f2i0osrmij511ou0t7rfhmnlsn-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cmr4pghcqg0403a5jq11dvfnlikl-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cmr4pghcqg0403a5jq11dvfnlikl-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cm85p376bf4l9af5geeqascs19re-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cm85p376bf4l9af5geeqascs19re-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cldmf4jan6oonnq2tg3nk9r52s81-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cldmf4jan6oonnq2tg3nk9r52s81-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-29T00:44:40Z Ended: 2026-10-29T00:55:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cm3bgo3iqkg6triqkq45fn7rdlr8-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cm3bgo3iqkg6triqkq45fn7rdlr8-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-29T00:45:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4dtdrbk41153d35r2v928ecfh6bmp-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4dtdrbk41153d35r2v928ecfh6bmp-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-29T00:45:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4dgi32u5j8fk020vol0qqv1ds1k11-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4dgi32u5j8fk020vol0qqv1ds1k11-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-29T00:45:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clhtd0hi1sub7sbhcb7dhg3pba48-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clhtd0hi1sub7sbhcb7dhg3pba48-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-29T00:45:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clv3ma8s477hmmkla42d167b93ol-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clv3ma8s477hmmkla42d167b93ol-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-29T00:46:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4davicql96oofkg045duq6nghl7ff-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4davicql96oofkg045duq6nghl7ff-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-29T00:46:40Z Ended: 2026-10-29T00:55:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4e557h8kaiucvu5gvvdsgqh4o581v-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4e557h8kaiucvu5gvvdsgqh4o581v-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-29T00:46:40Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ct1nqdjgi22uib0cecj6up04l0m7-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ct1nqdjgi22uib0cecj6up04l0m7-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clpeml5pupvinmlvbpse0ie0hfs9-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clpeml5pupvinmlvbpse0ie0hfs9-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4darhtjunfn21raucc3dfc59v09jp-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4darhtjunfn21raucc3dfc59v09jp-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ckvnln5c26n1qo12ulvfvcdqeadr-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ckvnln5c26n1qo12ulvfvcdqeadr-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4clafsmcnsm8b1l90dllklk71mg71-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4clafsmcnsm8b1l90dllklk71mg71-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4d88ealur630tj4sst5o6qkjn1dgl-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4d88ealur630tj4sst5o6qkjn1dgl-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:55:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4ctr3n2rtj0ed87918esf64n0iglc-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4ctr3n2rtj0ed87918esf64n0iglc-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4db5tooak0klk2r251u93t1sccjm0-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4db5tooak0klk2r251u93t1sccjm0-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-10-29T00:46:41Z Ended: 2026-10-29T00:56:00Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8902528-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2to4cl4qqobidbthts4n7r6dv1md7maq-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2to4cl4qqobidbthts4n7r6dv1md7maq-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (15) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | e2e | 14 | 3.38%14 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 23: 2 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 3 · Oct 28: 6 · Oct 29: 2 · Oct 30: 0 | INT, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x3674a9c4e220>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-t9dgnpvl98vn" to finish creating: GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34",
"name": "fc792b8b-e015-4d89-a0db-da7d13830c34",
"status": "Failed",
"startTime": "2026-10-29T16:16:06.194748972Z",
"endTime": "2026-10-29T16:45:10.710444768Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
correlation ID: 9f4e5eef-e698-490e-b8cf-ae403d05c79c
client request ID: 86598878-fce3-449c-9061-c59955db4018
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x3674a9c4e220>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-t9dgnpvl98vn" to finish creating: GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34",
"name": "fc792b8b-e015-4d89-a0db-da7d13830c34",
"status": "Failed",
"startTime": "2026-10-29T16:16:06.194748972Z",
"endTime": "2026-10-29T16:45:10.710444768Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3115648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc792b8b-e015-4d89-a0db-da7d13830c34
correlation ID: 9f4e5eef-e698-490e-b8cf-ae403d05c79c
client request ID: 86598878-fce3-449c-9061-c59955db4018
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x2091c74f61e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m676msdbxkvh" to finish creating: GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"name": "e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"status": "Failed",
"startTime": "2026-10-29T01:48:08.746106597Z",
"endTime": "2026-10-29T02:17:11.026048474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
correlation ID: 5c511849-df6c-49a5-a63d-d7d0d096b75d
client request ID: e632af41-8b08-40f4-bad1-f012e66c5cee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x2091c74f61e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m676msdbxkvh" to finish creating: GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"name": "e96f5f02-d572-47d2-bad4-ca27e9a80c54",
"status": "Failed",
"startTime": "2026-10-29T01:48:08.746106597Z",
"endTime": "2026-10-29T02:17:11.026048474Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9534592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e96f5f02-d572-47d2-bad4-ca27e9a80c54
correlation ID: 5c511849-df6c-49a5-a63d-d7d0d096b75d
client request ID: e632af41-8b08-40f4-bad1-f012e66c5cee
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1fcb88002b40>:
failed to create HCP cluster np-downgrade-llw24f: failed waiting for cluster="np-downgrade-llw24f" in resourcegroup="rg-np-downgrade-llw24f-kqx7lnhpdqpq" to finish creating: GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"name": "dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"status": "Failed",
"startTime": "2026-10-28T23:17:28.994096116Z",
"endTime": "2026-10-28T23:46:34.623847458Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
correlation ID: 162093f7-f0bd-4b1e-b072-d657b497ab59
client request ID: 0bb02caa-9eaa-4069-8399-efdeced8dac8
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1fcb88002b40>:
failed to create HCP cluster np-downgrade-llw24f: failed waiting for cluster="np-downgrade-llw24f" in resourcegroup="rg-np-downgrade-llw24f-kqx7lnhpdqpq" to finish creating: GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"name": "dcefa597-82c1-4b2e-9dc9-ff35d2de5251",
"status": "Failed",
"startTime": "2026-10-28T23:17:28.994096116Z",
"endTime": "2026-10-28T23:46:34.623847458Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3784704.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dcefa597-82c1-4b2e-9dc9-ff35d2de5251
correlation ID: 162093f7-f0bd-4b1e-b072-d657b497ab59
client request ID: 0bb02caa-9eaa-4069-8399-efdeced8dac8
occurredContributing tests (13)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodCrashLooping firedfull failure pattern: alert [hcp] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 13 | 3.14%13 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 23: 3 · Oct 24: 0 · Oct 25: 2 · Oct 26: 3 · Oct 27: 2 · Oct 28: 4 · Oct 29: 2 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:36:23Z Ended: 2026-10-29T17:06:22Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c0dd711d-3085-4fe4-ad56-0342b501a32c" Description: Pod ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-29T16:36:23Z Ended: 2026-10-29T17:05:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c0dd711d-3085-4fe4-ad56-0342b501a32c" Description: Pod ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:11:09Z Ended: 2026-10-29T02:38:06Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="13506377-a0a1-42df-b327-83b565596a02" Description: Pod ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-29T02:11:09Z Ended: 2026-10-29T02:39:08Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="13506377-a0a1-42df-b327-83b565596a02" Description: Pod ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:39:20Z Ended: 2026-10-28T21:05:19Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327" Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T20:40:18Z Ended: 2026-10-28T21:06:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327" Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (13) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [hcp] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 11 | 2.66%11 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 23: 3 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 2 · Oct 28: 4 · Oct 29: 2 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:49:22Z Ended: 2026-10-29T17:02:27Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-29T16:49:22Z Ended: 2026-10-29T17:02:27Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:24:07Z Ended: 2026-10-29T02:35:09Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-29T02:24:07Z Ended: 2026-10-29T02:35:09Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:52:20Z Ended: 2026-10-28T21:03:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:52:20Z Ended: 2026-10-28T21:03:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 11 | 2.66%11 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 23: 2 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 2 · Oct 28: 4 · Oct 29: 2 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:34:27Z Ended: 2026-10-29T17:02:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-29T16:45:26Z Ended: 2026-10-29T17:01:23Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3115648-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.110:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3115648/providers/microsoft.monitor/accounts/hcps-j3115648", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster", pod="arohcp-monitor-kube-state-metrics-77c9cb8f68-6t4t4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2toi6hgjcichbvg928v4qi1bj3hqpom9-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:09:08Z Ended: 2026-10-29T02:34:07Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-29T02:09:08Z Ended: 2026-10-29T02:34:07Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9534592-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.13:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9534592/providers/microsoft.monitor/accounts/hcps-j9534592", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-gcfwn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to5fmfhh0f49io7rqjq1ithbotsvf13-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:37:24Z Ended: 2026-10-28T21:04:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:37:24Z Ended: 2026-10-28T21:01:24Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | alert | 11 | 2.66%11 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 23: 1 · Oct 24: 2 · Oct 25: 0 · Oct 26: 0 · Oct 27: 6 · Oct 28: 1 · Oct 29: 2 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 3 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T12:12:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2007552-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-9889fbcdb-wfqxr", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-9889fbcdb-wfqxr has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-29T12:12:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2007552-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-9889fbcdb-6cndc", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-9889fbcdb-6cndc has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-29T12:13:39Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2007552-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-85d5f7ccc-bw7nz", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-85d5f7ccc-bw7nz has been in a non-ready state for longer than 5 minutes. alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6974c85f45-zzrbj", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-6974c85f45-zzrbj has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6974c85f45-p5jmb", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-6974c85f45-p5jmb has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-7cbc64f544-mxsh9", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-7cbc64f544-mxsh9 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-29T03:24:02Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0081024-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6974c85f45-lfvfw", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-6974c85f45-lfvfw has been in a non-ready state for longer than 5 minutes. alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-64b56768b9-r7v9d", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-64b56768b9-r7v9d has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-7b9596cccb-g9thf", region="canadacentral", severity="warning" Description: Pod fleet/fleet-controller-7b9596cccb-g9thf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-t9snj", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-t9snj has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-rm24l", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-rm24l has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | e2e | 10 | 2.42%10 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 2 · Oct 26: 0 · Oct 27: 4 · Oct 28: 1 · Oct 29: 3 · Oct 30: 0 | INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x2e578d11a500>:
DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2e578d07c700>{
msg: "DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x2e578d11a500>:
DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2e578d07c700>{
msg: "DNS for agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vj9nr.apps.aro.cluster.gtffc.j8011904.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-kns88h"
Unexpected error:
<*fmt.wrapError | 0x35a5d6ab6c80>:
DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x35a5d645c1c0>{
msg: "DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-kns88h"
Unexpected error:
<*fmt.wrapError | 0x35a5d6ab6c80>:
DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x35a5d645c1c0>{
msg: "DNS for agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qtff.apps.aro.cluster.6clxs.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0x1942e140ed80>:
DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1942e1526600>{
msg: "DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0x1942e140ed80>:
DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1942e1526600>{
msg: "DNS for agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7sgh5.apps.aro.cluster.lsqg8.j7470080.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (9)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | alert | 10 | 2.42%10 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 23: 1 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 5 · Oct 28: 2 · Oct 29: 2 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T20:25:29Z Ended: 2026-10-29T20:31:30Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4005376-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T02:21:29Z Ended: 2026-10-29T02:39:31Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7406208-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T21:09:13Z Ended: 2026-10-28T21:26:12Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8906368-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | e2e | 8 | 1.93%8 of 414 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 4 · Oct 27: 3 · Oct 28: 1 · Oct 29: 2 · Oct 30: 0 | PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3578474c0240>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gjfz7z4nxgmg" to finish creating: GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"name": "d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"status": "Failed",
"startTime": "2026-10-29T21:48:35.303274184Z",
"endTime": "2026-10-29T22:07:37.57729809Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-5ww8k-lpn65: NodeDoesNotExist: Node does not exist\nMachine cluster-np-1-5ww8k-56zrg: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-5ww8k-lpn65: Failed: virtualmachine failed to create or update. err: failed to get existing resource ingress-cert-le-gjfz7z4nxgmg--managed/cluster-np-1-5ww8k-lpn65 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gjfz7z4nxgmg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5ww8k-lpn65\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-thro... message truncated\nMachine cluster-np-1-5ww8k-56zrg: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
correlation ID: 443d1b16-e3d5-463c-b730-8cd4c86e6293
client request ID: 2bd9ac40-1ceb-4e60-80b5-16c2651e38ce
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3578474c0240>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gjfz7z4nxgmg" to finish creating: GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"name": "d86719cc-79fc-4a67-9af1-1c8ca9d4719e",
"status": "Failed",
"startTime": "2026-10-29T21:48:35.303274184Z",
"endTime": "2026-10-29T22:07:37.57729809Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-5ww8k-lpn65: NodeDoesNotExist: Node does not exist\nMachine cluster-np-1-5ww8k-56zrg: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-5ww8k-lpn65: Failed: virtualmachine failed to create or update. err: failed to get existing resource ingress-cert-le-gjfz7z4nxgmg--managed/cluster-np-1-5ww8k-lpn65 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-gjfz7z4nxgmg--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-5ww8k-lpn65\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'fd35ac5f-6493-40cb-9131-d669a70114c3' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-thro... message truncated\nMachine cluster-np-1-5ww8k-56zrg: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3760640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d86719cc-79fc-4a67-9af1-1c8ca9d4719e
correlation ID: 443d1b16-e3d5-463c-b730-8cd4c86e6293
client request ID: 2bd9ac40-1ceb-4e60-80b5-16c2651e38cefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x6f3e064d560>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-nw86wtvvh485" to finish creating: GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"name": "a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"status": "Failed",
"startTime": "2026-10-29T11:24:38.055894126Z",
"endTime": "2026-10-29T11:43:42.222979982Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-fjfm9-tkhpd: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-fjfm9-tkhpd: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-nw86wtvvh485--managed/cluster-np-1-fjfm9-tkhpd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-nw86wtvvh485--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-fjfm9-tkhpd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
correlation ID: d21776c7-10dd-4392-956c-91ea278d84d3
client request ID: 0326f865-16a8-401d-88b7-c15ecd702d5e
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x6f3e064d560>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-nw86wtvvh485" to finish creating: GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"name": "a4a22bf9-a88d-42da-a639-fe84834ff4c0",
"status": "Failed",
"startTime": "2026-10-29T11:24:38.055894126Z",
"endTime": "2026-10-29T11:43:42.222979982Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-fjfm9-tkhpd: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-fjfm9-tkhpd: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-nw86wtvvh485--managed/cluster-np-1-fjfm9-tkhpd-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-nw86wtvvh485--managed/providers/Microsoft.Network/networkInterfaces/cluster-np-1-fjfm9-tkhpd-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3015be55-a361-4a86-8439-0abc7860a4ef' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-th... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7399680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4a22bf9-a88d-42da-a639-fe84834ff4c0
correlation ID: d21776c7-10dd-4392-956c-91ea278d84d3
client request ID: 0326f865-16a8-401d-88b7-c15ecd702d5efail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549Contributing tests (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
at least one schedule should have a LastBackupTimefull failure pattern: at least one schedule should have a LastBackupTime Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 23 | 5.56%23 of 414 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 22 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | alert | 20 | 4.83%20 of 414 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | Show trend detailsOct 23: 3 · Oct 24: 2 · Oct 25: 4 · Oct 26: 4 · Oct 27: 2 · Oct 28: 4 · Oct 29: 4 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T19:16:57Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.225:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-nc7qt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dbf8e658-1bc7-430e-8ce2-1f6c76f62e38" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-nc7qt (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-29T19:19:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.57:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-kml94", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8b51f9f0-c076-4060-b6fb-4a9e1396b5b6" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-kml94 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-29T19:19:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.225:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-nc7qt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dbf8e658-1bc7-430e-8ce2-1f6c76f62e38" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-nc7qt (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-29T19:19:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0994944-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.57:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0994944/providers/microsoft.monitor/accounts/services-j0994944", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-kml94", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8b51f9f0-c076-4060-b6fb-4a9e1396b5b6" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-kml94 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T16:05:53Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-rdnhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="88524800-d6ff-4cb4-8768-11325673da1b" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-rdnhz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-29T16:07:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.229:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-6v8bj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cc2859a5-f39a-4c88-a623-ff48298d4334" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-6v8bj (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-29T16:07:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-rdnhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="88524800-d6ff-4cb4-8768-11325673da1b" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-rdnhz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-29T16:07:52Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5155584-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.229:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5155584/providers/microsoft.monitor/accounts/services-j5155584", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-6454dbbb5b-6v8bj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cc2859a5-f39a-4c88-a623-ff48298d4334" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-6454dbbb5b-6v8bj (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T08:05:53Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-87fq6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fb1a32bd-f4c6-4a27-a3e9-8ea2cd5b63a8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-87fq6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-29T08:06:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.246:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-xfr2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0d3b843f-d25e-4240-979b-c8cbbb202083" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-xfr2d (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-29T08:08:55Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.246:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-xfr2d", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0d3b843f-d25e-4240-979b-c8cbbb202083" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-xfr2d (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-29T08:09:54Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6803712-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6803712/providers/microsoft.monitor/accounts/services-j6803712", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-5ccb684f7d-87fq6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="fb1a32bd-f4c6-4a27-a3e9-8ea2cd5b63a8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-5ccb684f7d-87fq6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (20) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | provision | 10 | 2.42%10 of 414 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 2 · Oct 27: 4 · Oct 28: 3 · Oct 29: 1 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-29T04:30:23.876Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T04:30:25.398Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T04:30:26.536Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f45d1e5c6b865770f920f060f41efd0137429ae403f880e6cf3c7dda4bb749b3 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1611392-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff45d1e5c6b865770f920f060f41efd0137429ae403f880e6cf3c7dda4bb749b3
time=2026-10-29T04:32:57.126Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.Resources/deployments/f45d1e5c6b865770f920f060f41efd0137429ae403f880e6cf3c7dda4bb749b3/operationStatuses/08584083586595300523\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b9d1b967-fce3-4291-995e-41596346ad76\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.Resources/deployments/cluster-fjmdbyledblew\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b9d1b967-fce3-4291-995e-41596346ad76\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j1611392-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j1611392-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-29T04:34:57.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T04:34:58.130Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T04:35:07.065Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=19b7db859db524fa83889a05d098c66f5305747330cca9423e2ad72279dea19a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1611392-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F19b7db859db524fa83889a05d098c66f5305747330cca9423e2ad72279dea19a
time=2026-10-29T04:36:37.470Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1611392-svc/providers/Microsoft.Resources/deployments/19b7db859db524fa83889a05d098c66f5305747330cca9423e2ad72279dea19a/operationStatuses/08584083583803398033\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: df15aba7-50a9-49f1-a665-fa25966a7d0e\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-29T00:10:51.309Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T00:10:53.106Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T00:10:54.792Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4319744-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4
time=2026-10-29T00:12:55.689Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/3eb04a406861dadce17733f3106b46e7d01c706185214584bf436d2dff1602d4/operationStatuses/08584083742316437961\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/cluster-6um4nhqkdznjc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j4319744-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j4319744-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-29T00:14:55.689Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-29T00:14:56.876Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-29T00:14:58.481Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4319744-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45
time=2026-10-29T00:16:28.867Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4319744-svc/providers/Microsoft.Resources/deployments/3d3c29485896acfd7e15d9c09cc1476e363573f3512d196c907c860f06655a45/operationStatuses/08584083739878467732\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-28T16:43:28.511Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:43:31.273Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:43:32.448Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87
time=2026-10-28T16:45:32.811Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87/operationStatuses/08584084010736719502\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/cluster-rybldmoz73lrm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2862080-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2862080-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T16:47:32.817Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:47:34.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:47:35.068Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006
time=2026-10-28T16:49:05.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006/operationStatuses/08584084008309075646\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | e2e | 8 | 1.93%8 of 414 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsOct 23: 2 · Oct 24: 3 · Oct 25: 0 · Oct 26: 1 · Oct 27: 2 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 6 | 1.45%6 of 414 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 23: 2 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x1ddd7d2831b8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zk7rbqmsp2zd" to finish creating: GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"name": "cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"status": "Failed",
"startTime": "2026-10-29T00:42:47.491540846Z",
"endTime": "2026-10-29T01:01:51.350365806Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-ztzhx-l97nr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-ztzhx-l97nr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cilium-np-ztzhx-547rj: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-zk7rbqmsp2zd--managed/cluster-cilium-np-ztzhx-547rj-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Network/networkInterfaces/cluster-cilium-np-ztzhx-547rj-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/a... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
correlation ID: 52dc9816-9bf4-4c02-9a62-7105502ff251
client request ID: 3591f936-ad0d-47b0-b46a-a9f82bf4e6ae
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x1ddd7d2831b8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-zk7rbqmsp2zd" to finish creating: GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"name": "cb6f6658-c768-40a6-b3cf-8a5bd044305f",
"status": "Failed",
"startTime": "2026-10-29T00:42:47.491540846Z",
"endTime": "2026-10-29T01:01:51.350365806Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-ztzhx-l97nr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-ztzhx-l97nr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-cilium-np-ztzhx-547rj: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-zk7rbqmsp2zd--managed/cluster-cilium-np-ztzhx-547rj-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zk7rbqmsp2zd--managed/providers/Microsoft.Network/networkInterfaces/cluster-cilium-np-ztzhx-547rj-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/a... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8902528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6f6658-c768-40a6-b3cf-8a5bd044305f
correlation ID: 52dc9816-9bf4-4c02-9a62-7105502ff251
client request ID: 3591f936-ad0d-47b0-b46a-a9f82bf4e6ae
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurredContributing tests (3)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 5 | 1.21%5 of 414 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 5 · Oct 30: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-8pc9jb"
Unexpected error:
<*errors.joinError | 0x19fbe63ec288>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-8pc9jb-xk8t4-5mzft": nodes "cluster-np-8pc9jb-xk8t4-5mzft" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x19fbe659e180>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e160>{
msg: "failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e140>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x19fbe658ea00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-8pc9jb-xk8t4-5mzft",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-8pc9jb"
Unexpected error:
<*errors.joinError | 0x19fbe63ec288>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-8pc9jb-xk8t4-5mzft": nodes "cluster-np-8pc9jb-xk8t4-5mzft" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x19fbe659e180>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e160>{
msg: "failed to label node cluster-np-8pc9jb-xk8t4-5mzft egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x19fbe659e140>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-8pc9jb-xk8t4-5mzft\": nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x19fbe658ea00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-8pc9jb-xk8t4-5mzft\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-8pc9jb-xk8t4-5mzft",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-spgbv9"
Unexpected error:
<*errors.joinError | 0xb85fe22660>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-spgbv9-bvtl9-fgbn6": nodes "cluster-np-spgbv9-bvtl9-fgbn6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xb8604081a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408180>{
msg: "failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408160>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xb860403860>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-spgbv9-bvtl9-fgbn6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-spgbv9"
Unexpected error:
<*errors.joinError | 0xb85fe22660>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-spgbv9-bvtl9-fgbn6": nodes "cluster-np-spgbv9-bvtl9-fgbn6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xb8604081a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408180>{
msg: "failed to label node cluster-np-spgbv9-bvtl9-fgbn6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xb860408160>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-spgbv9-bvtl9-fgbn6\": nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xb860403860>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-spgbv9-bvtl9-fgbn6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-spgbv9-bvtl9-fgbn6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-j2tgkf"
Unexpected error:
<*errors.joinError | 0xf595dece090>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-j2tgkf-q8c9h-j47g6": nodes "cluster-np-j2tgkf-q8c9h-j47g6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xf595db0f4c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f4a0>{
msg: "failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f480>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xf595e31e6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-j2tgkf-q8c9h-j47g6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-j2tgkf"
Unexpected error:
<*errors.joinError | 0xf595dece090>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-j2tgkf-q8c9h-j47g6": nodes "cluster-np-j2tgkf-q8c9h-j47g6" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xf595db0f4c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f4a0>{
msg: "failed to label node cluster-np-j2tgkf-q8c9h-j47g6 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xf595db0f480>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-j2tgkf-q8c9h-j47g6\": nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xf595e31e6e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-j2tgkf-q8c9h-j47g6\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-j2tgkf-q8c9h-j47g6",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] hypershiftOperatorPodsDown firedfull failure pattern: alert [svc] hypershiftOperatorPodsDown fired Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | alert | 5 | 1.21%5 of 414 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 4 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T16:11:35Z Ended: 2026-10-29T16:17:37Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j8756352-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j8756352-mgmt-2 alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-29T14:55:08Z Ended: 2026-10-29T15:01:06Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j0084992-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j0084992-mgmt-2 alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-29T12:14:15Z Severity: Sev2 Labels: alertname="hypershiftOperatorPodsDown", cluster="ci01-j2007552-mgmt-2", component="slo", service="hypershift-operator", severity="2", slo="hypershift-operator-availability" Description: 2 hypershift-operator pods have been unavailable for 5 minutes. Management Cluster: ci01-j2007552-mgmt-2 Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context de...full failure pattern: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | provision | 5 | 1.21%5 of 414 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 4 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-27T15:06:00.351Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-10-27T15:06:00.927Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:72abeb657cfbdd6a5891fc102a16429875d70c5932ad5994524d0246db914013 imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:55a0ad895a8f10193bb070883194449f5252c057353887af1ba0767760a0d2f0 imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-27T15:06:00.927Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-27T15:06:01.327Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller time=2026-10-27T15:06:01.636Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-10-27T15:06:01.636Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-27T15:06:01.765Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-27T15:11:07.858Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-27T15:11:07.859Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T10:27:22.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-10-25T10:27:22.644Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T10:27:22.644Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:22.918Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller time=2026-10-25T10:27:23.279Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-10-25T10:27:23.279Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:23.445Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:32:28.984Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T10:32:28.985Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:14:44.299Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T08:14:45.051Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T08:14:45.051Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:45.565Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T08:14:45.984Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T08:14:45.984Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:46.156Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:19:54.254Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:19:54.254Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" Contributing tests (2)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] ClusterCredentialMetricMissing firedfull failure pattern: alert [svc] ClusterCredentialMetricMissing fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | alert | 5 | 1.21%5 of 414 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T01:28:46Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j5915264-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j5915264-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T21:37:46Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j3842944-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j3842944-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T18:18:42Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j4499328-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j4499328-svc. Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 7 | 18.42%7 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 5 · Oct 24: 7 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-l96rfk"
Unexpected error:
<*fmt.wrapError | 0x11cfecaf8760>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x11cfee5e70e0>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x11cfee5e70b0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x11cfee823170>{
N: {
neg: false,
abs: [12328844732508617451, 3386053924852305530, 11508787352079574519, 580165219016783073, 15126087406076015328, 8473747687808762055, 6638534397921598209, 4531117369475629035, 8686734636532653775, 9699186360867822250, 15005114980278798855, 3900631334283717631, 15336487295176330797, 13597435546774256175, 1139063161098112232, 10161195232013165903, 15551291529941624294, 8053304939740217029, 18299432943387333464, 12035498982612883405, 1076299365375342957, 3325077180629581387, 4778720590180299818, 5926909629444315306, 16602979226450151451, 14838171221845856625, 18308053321550668292, 3951715190352458341, 11733219666321171503, 11793477282006375930, 17943060274730047763, 12841897204482256645],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8845870848929380208],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928477681, loc: nil},
NotAfter: {wall: 0, ext: 63960013681, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
AuthorityKeyId: [248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x11cfed040008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-l96rfk"
Unexpected error:
<*fmt.wrapError | 0x11cfecaf8760>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x11cfee5e70e0>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x11cfee5e70b0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x11cfee823170>{
N: {
neg: false,
abs: [12328844732508617451, 3386053924852305530, 11508787352079574519, 580165219016783073, 15126087406076015328, 8473747687808762055, 6638534397921598209, 4531117369475629035, 8686734636532653775, 9699186360867822250, 15005114980278798855, 3900631334283717631, 15336487295176330797, 13597435546774256175, 1139063161098112232, 10161195232013165903, 15551291529941624294, 8053304939740217029, 18299432943387333464, 12035498982612883405, 1076299365375342957, 3325077180629581387, 4778720590180299818, 5926909629444315306, 16602979226450151451, 14838171221845856625, 18308053321550668292, 3951715190352458341, 11733219666321171503, 11793477282006375930, 17943060274730047763, 12841897204482256645],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8845870848929380208],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928477681, loc: nil},
NotAfter: {wall: 0, ext: 63960013681, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
AuthorityKeyId: [248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x11cfed040008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-2rcw86"
Unexpected error:
<*fmt.wrapError | 0x2b2e578984a0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x2b2e57370780>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x2b2e57370750>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x2b2e57843d70>{
N: {
neg: false,
abs: [2877128800215149205, 16629919685921545055, 13732506504615110447, 11103748926260740575, 11416989105297106818, 9223988340828442177, 2897202458616478060, 11766261893188694937, 15086238765284949963, 10809216967110507236, 9979912210168109780, 11664503868611822661, 3024932991847888604, 17852206095763650445, 1698336212534998063, 17691123283498957122, 9751057521779199143, 5748498642685419890, 4821085446021960184, 2270194980282856385, 5998191060056422898, 1979344939107882427, 11002240530182040303, 17961926253159653156, 4444200366252054316, 17136494472856312874, 5046575580329891885, 18333653388545728381, 18332734810168546880, 2830450700003086872, 11572718067588258363, 13260460152132755496],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8959707475658302546],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928472888, loc: nil},
NotAfter: {wall: 0, ext: 63960008888, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
AuthorityKeyId: [145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x2b2e55ba2008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-2rcw86"
Unexpected error:
<*fmt.wrapError | 0x2b2e578984a0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x2b2e57370780>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x2b2e57370750>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x2b2e57843d70>{
N: {
neg: false,
abs: [2877128800215149205, 16629919685921545055, 13732506504615110447, 11103748926260740575, 11416989105297106818, 9223988340828442177, 2897202458616478060, 11766261893188694937, 15086238765284949963, 10809216967110507236, 9979912210168109780, 11664503868611822661, 3024932991847888604, 17852206095763650445, 1698336212534998063, 17691123283498957122, 9751057521779199143, 5748498642685419890, 4821085446021960184, 2270194980282856385, 5998191060056422898, 1979344939107882427, 11002240530182040303, 17961926253159653156, 4444200366252054316, 17136494472856312874, 5046575580329891885, 18333653388545728381, 18332734810168546880, 2830450700003086872, 11572718067588258363, 13260460152132755496],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8959707475658302546],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928472888, loc: nil},
NotAfter: {wall: 0, ext: 63960008888, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
AuthorityKeyId: [145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x2b2e55ba2008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-wk4gjj"
Unexpected error:
<*fmt.wrapError | 0x30fb7a9c6d80>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x30fb7cb9b320>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x30fb7cb9b2f0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x30fb7a9895b0>{
N: {
neg: false,
abs: [3713275565900749005, 5533862720488119315, 886043176200789153, 17818264461995138607, 16265450768615453130, 5834472357489801107, 5862144673391444614, 628656117149906395, 13178964295004987606, 715618696217184398, 12066541168516629782, 6050057140458382287, 14400619811794133371, 6862502860246136883, 1422013248764323211, 8372522752467421288, 13138907997452357860, 17826693266249031145, 5098815638656127509, 5738821655630622400, 12438080729488309683, 6057798802862875801, 7503648299216627878, 13557164215630681915, 9649412887563066602, 17200267080417426931, 12116969928591911697, 17745395029646614848, 8889710084128733264, 16509317662742049607, 11507875977146301207, 12660817078547036528],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [7949786151822990365],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928459200, loc: nil},
NotAfter: {wall: 0, ext: 63959995200, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
AuthorityKeyId: [107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x30fb7a8f2c08 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-wk4gjj"
Unexpected error:
<*fmt.wrapError | 0x30fb7a9c6d80>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x30fb7cb9b320>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x30fb7cb9b2f0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x30fb7a9895b0>{
N: {
neg: false,
abs: [3713275565900749005, 5533862720488119315, 886043176200789153, 17818264461995138607, 16265450768615453130, 5834472357489801107, 5862144673391444614, 628656117149906395, 13178964295004987606, 715618696217184398, 12066541168516629782, 6050057140458382287, 14400619811794133371, 6862502860246136883, 1422013248764323211, 8372522752467421288, 13138907997452357860, 17826693266249031145, 5098815638656127509, 5738821655630622400, 12438080729488309683, 6057798802862875801, 7503648299216627878, 13557164215630681915, 9649412887563066602, 17200267080417426931, 12116969928591911697, 17745395029646614848, 8889710084128733264, 16509317662742049607, 11507875977146301207, 12660817078547036528],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [7949786151822990365],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928459200, loc: nil},
NotAfter: {wall: 0, ext: 63959995200, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
AuthorityKeyId: [107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x30fb7a8f2c08 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (7)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | unknown | 4 | 10.53%4 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sl7v6jcfzpzn/cluster-candidate-5-0-rw67x7 should provision
Unexpected error:
<*fmt.wrapError | 0x320a55350300>:
failed to create HCP cluster cluster-candidate-5-0-rw67x7: failed waiting for cluster="cluster-candidate-5-0-rw67x7" in resourcegroup="rg-candidate-5-0-sl7v6jcfzpzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"name": "e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"status": "Failed",
"startTime": "2026-10-29T01:24:15.287612604Z",
"endTime": "2026-10-29T01:53:18.05722195Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
correlation ID: c2102baf-9a88-4da3-a6f8-854193b8aaa9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-sl7v6jcfzpzn/cluster-candidate-5-0-rw67x7 should provision
Unexpected error:
<*fmt.wrapError | 0x320a55350300>:
failed to create HCP cluster cluster-candidate-5-0-rw67x7: failed waiting for cluster="cluster-candidate-5-0-rw67x7" in resourcegroup="rg-candidate-5-0-sl7v6jcfzpzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"name": "e0c6d8e2-15ce-4865-b6bb-324c45a5ce23",
"status": "Failed",
"startTime": "2026-10-29T01:24:15.287612604Z",
"endTime": "2026-10-29T01:53:18.05722195Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e0c6d8e2-15ce-4865-b6bb-324c45a5ce23
correlation ID: c2102baf-9a88-4da3-a6f8-854193b8aaa9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 2 | 5.26%2 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | PROD | |||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x248e56e12588>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x248e56f14360>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e5757aa00>{
msg: "DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e57018e60>{
msg: "DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x248e56e12588>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x248e56f14360>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e5757aa00>{
msg: "DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e57018e60>{
msg: "DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x22d4bdfbc060>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x22d4bdf9c1a0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdee0f00>{
msg: "DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdf9da20>{
msg: "DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x22d4bdfbc060>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x22d4bdf9c1a0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdee0f00>{
msg: "DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdf9da20>{
msg: "DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 2 | 5.26%2 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0xa204c386ab0>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-zhz5sxs2r4gt" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-zhz5sxs2r4gt\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xa204b336550>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0xa204c386ab0>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-zhz5sxs2r4gt" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-zhz5sxs2r4gt\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xa204b336550>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 1 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0xdad4da81e80>:
DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdad4d9b41a0>{
msg: "DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0xdad4da81e80>:
DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdad4d9b41a0>{
msg: "DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:118]: failed to create default nodepool np-1
Unexpected error:
<*fmt.wrapError | 0x3cdcfbca5680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-8vtwfvfssx9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0",
"name": "907ac069-8701-4021-b410-cf10181345c0",
"status": "Failed",
"startTime": "2026-10-25T05:14:31.336281353Z",
"endTime": "2026-10-25T05:33:33.578871475Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-p7hpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: Failed: interfaces failed to create or update. err: failed to get existing resource rg-gpu-nodepool-8vtwfvfssx9k--managed/cluster-gpu-np-np-1-8v7xk-p7hpl-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Network/networkInterfaces/cluster-gpu-np-np-1-8v7xk-p7hpl-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to ... message truncated\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
correlation ID: 1c529a86-2bdd-43d7-ab0e-4267cd216305
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:118]: failed to create default nodepool np-1
Unexpected error:
<*fmt.wrapError | 0x3cdcfbca5680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-8vtwfvfssx9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0",
"name": "907ac069-8701-4021-b410-cf10181345c0",
"status": "Failed",
"startTime": "2026-10-25T05:14:31.336281353Z",
"endTime": "2026-10-25T05:33:33.578871475Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-p7hpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: Failed: interfaces failed to create or update. err: failed to get existing resource rg-gpu-nodepool-8vtwfvfssx9k--managed/cluster-gpu-np-np-1-8v7xk-p7hpl-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Network/networkInterfaces/cluster-gpu-np-np-1-8v7xk-p7hpl-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to ... message truncated\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
correlation ID: 1c529a86-2bdd-43d7-ab0e-4267cd216305
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x3f57e0004820>:
DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x3f57dfc6e000>{
msg: "DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x3f57e0004820>:
DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x3f57dfc6e000>{
msg: "DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-<id>full failure pattern: DNS for route host agnhost-e2e-sample-app-<id> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x7f30d39c620>:
DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f30d39c600>{
msg: "DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x7f30d39c620>:
DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f30d39c600>{
msg: "DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Ple... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf"
Unexpected error:
<*fmt.wrapError | 0x3b484fe8ea00>:
failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf" in resourcegroup="rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"name": "cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"status": "Failed",
"startTime": "2026-10-24T03:36:53.315230765Z",
"endTime": "2026-10-24T03:56:02.496643822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j: Failed: interfaces failed to create or update. err: failed to get existing resource rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/providers/Microsoft.Network/networkInterfaces/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Ple... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
correlation ID: 3b16cd17-d78f-4b89-a83b-76ef2e36cf89
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf"
Unexpected error:
<*fmt.wrapError | 0x3b484fe8ea00>:
failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf" in resourcegroup="rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"name": "cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"status": "Failed",
"startTime": "2026-10-24T03:36:53.315230765Z",
"endTime": "2026-10-24T03:56:02.496643822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j: Failed: interfaces failed to create or update. err: failed to get existing resource rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/providers/Microsoft.Network/networkInterfaces/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Ple... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
correlation ID: 3b16cd17-d78f-4b89-a83b-76ef2e36cf89
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x9134b719420>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-c2zb7x4tvbjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"name": "2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"status": "Failed",
"startTime": "2026-10-24T17:16:34.851278228Z",
"endTime": "2026-10-24T17:45:35.221810696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
correlation ID: 4e6d7a8e-ea3e-4773-abae-818d5133da86
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x9134b719420>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-c2zb7x4tvbjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"name": "2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"status": "Failed",
"startTime": "2026-10-24T17:16:34.851278228Z",
"endTime": "2026-10-24T17:45:35.221810696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
correlation ID: 4e6d7a8e-ea3e-4773-abae-818d5133da86
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-c2hldx' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-c2hldx' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-5-1-g8n765"
Unexpected error:
<*errors.joinError | 0x254aca410498>:
VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.008s: DNS A record "*.e2e-dns-c2hldx" not yet present in zone rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-c2hldx' does not exist in resource group 'rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
correlation ID: 192cc89e-15d5-4e23-8532-45b3cbb07059
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-5-1-g8n765"
Unexpected error:
<*errors.joinError | 0x254aca410498>:
VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.008s: DNS A record "*.e2e-dns-c2hldx" not yet present in zone rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-c2hldx' does not exist in resource group 'rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
correlation ID: 192cc89e-15d5-4e23-8532-45b3cbb07059
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-jgd8x5' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-jgd8x5' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-652kdm"
Unexpected error:
<*errors.joinError | 0x282c0f144570>:
VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.006s: DNS A record "*.e2e-dns-jgd8x5" not yet present in zone rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-jgd8x5' does not exist in resource group 'rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
correlation ID: a77d314a-799e-4560-8840-6738610cf0af
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-652kdm"
Unexpected error:
<*errors.joinError | 0x282c0f144570>:
VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.006s: DNS A record "*.e2e-dns-jgd8x5" not yet present in zone rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-jgd8x5' does not exist in resource group 'rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
correlation ID: a77d314a-799e-4560-8840-6738610cf0af
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rg99mn' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rg99mn' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-4fhvgp"
Unexpected error:
<*errors.joinError | 0x3188fb29e5a0>:
VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.005s: DNS A record "*.e2e-dns-rg99mn" not yet present in zone rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-rg99mn' does not exist in resource group 'rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
correlation ID: cfd4563d-f15e-486d-ad09-c0f336987b6a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-4fhvgp"
Unexpected error:
<*errors.joinError | 0x3188fb29e5a0>:
VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.005s: DNS A record "*.e2e-dns-rg99mn" not yet present in zone rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-rg99mn' does not exist in resource group 'rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
correlation ID: cfd4563d-f15e-486d-ad09-c0f336987b6a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
Internal error occurred: failed calling webhook "webhookfull failure pattern: Internal error occurred: failed calling webhook "webhook Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:372]: Timed out after 120.003s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
<*errors.StatusError | 0x11d151799220>:
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Reason: "InternalError",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "",
Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:372]: Timed out after 120.003s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
<*errors.StatusError | 0x11d151799220>:
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Reason: "InternalError",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "",
Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
Process did not finish before 5m0s timeoutfull failure pattern: Process did not finish before 5m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 | DEV | |||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 5m0s timeout","severity":"error","time":"2026-10-29T19:59:31Z"}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.63%1 of 38 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 | DEV | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x866rm"
Unexpected error:
<*errors.joinError | 0x2cfb1b0a33e0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-x866rm-lmp2b-9wggz": nodes "cluster-np-x866rm-lmp2b-9wggz" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2cfb1c1b05e0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05c0>{
msg: "failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05a0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2cfb1c1b6460>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-x866rm-lmp2b-9wggz",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x866rm"
Unexpected error:
<*errors.joinError | 0x2cfb1b0a33e0>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-x866rm-lmp2b-9wggz": nodes "cluster-np-x866rm-lmp2b-9wggz" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2cfb1c1b05e0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05c0>{
msg: "failed to label node cluster-np-x866rm-lmp2b-9wggz egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2cfb1c1b05a0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-x866rm-lmp2b-9wggz\": nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2cfb1c1b6460>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-x866rm-lmp2b-9wggz\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-x866rm-lmp2b-9wggz",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 54.55% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 54.55% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 63.64% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 72.73% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 72.73% | 11 | stage/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 3 | 10.71%3 of 28 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 23: 9 · Oct 24: 5 · Oct 25: 0 · Oct 26: 2 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (8)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment | e2e | 1 | 3.57%1 of 28 job runs affected | NoiseSignal: Noise — struct/object fragment | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-h45rtr"
Expected success, but got an error:
<*fmt.wrapError | 0x31f100c0b6e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x31f100c0b6c0>{
observation: <*errors.errorString | 0x31f1027391b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x31f100c0b6a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-h45rtr"
Expected success, but got an error:
<*fmt.wrapError | 0x31f100c0b6e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x31f100c0b6c0>{
observation: <*errors.errorString | 0x31f1027391b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x31f100c0b6a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 3 | 10.71%3 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 23: 10 · Oct 24: 4 · Oct 25: 0 · Oct 26: 2 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (6)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 10 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | DEV, INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-vrx699" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xb37c723e620>:
failed to create HCP cluster cluster-zstream-4-22-vrx699: failed waiting for cluster="cluster-zstream-4-22-vrx699" in resourcegroup="rg-zstream-upgrade-4-22-mk5975tqzhdb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa",
"name": "f522dee9-4911-4a39-a8b7-66d53ae810aa",
"status": "Failed",
"startTime": "2026-10-28T21:21:44.905508951Z",
"endTime": "2026-10-28T21:50:49.351811692Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
correlation ID: fa068c44-117e-4214-8c6d-206698ccb0ad
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-vrx699" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xb37c723e620>:
failed to create HCP cluster cluster-zstream-4-22-vrx699: failed waiting for cluster="cluster-zstream-4-22-vrx699" in resourcegroup="rg-zstream-upgrade-4-22-mk5975tqzhdb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa",
"name": "f522dee9-4911-4a39-a8b7-66d53ae810aa",
"status": "Failed",
"startTime": "2026-10-28T21:21:44.905508951Z",
"endTime": "2026-10-28T21:50:49.351811692Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f522dee9-4911-4a39-a8b7-66d53ae810aa
correlation ID: fa068c44-117e-4214-8c6d-206698ccb0ad
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not existfull failure pattern: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x1ea319f5c1c8>:
resource group "rg-nightly-5-0-zf68j445hrrz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-5-0-md89qh" in resourcegroup="rg-nightly-5-0-zf68j445hrrz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1",
"name": "1895187e-5536-47a1-a44d-a9b6da901fa1",
"status": "Failed",
"startTime": "2026-10-25T00:01:49.583919819Z",
"endTime": "2026-10-25T00:25:49.807200592Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
correlation ID: 78c885d0-28f3-4f61-bdbb-0c2134f660b2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x1ea319f5c1c8>:
resource group "rg-nightly-5-0-zf68j445hrrz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-5-0-md89qh" in resourcegroup="rg-nightly-5-0-zf68j445hrrz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1",
"name": "1895187e-5536-47a1-a44d-a9b6da901fa1",
"status": "Failed",
"startTime": "2026-10-25T00:01:49.583919819Z",
"endTime": "2026-10-25T00:25:49.807200592Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
correlation ID: 78c885d0-28f3-4f61-bdbb-0c2134f660b2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCustomIngressCertificate timed out after 10m0full failure pattern: VerifyCustomIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:525]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x36a1186625c0>:
VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*fmt.wrapError | 0x36a1186623e0>{
msg: "ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x36a118d74840>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
Err: <*tls.CertificateVerificationError | 0x36a118d74810>{
UnverifiedCertificates: [
{
Raw: [48, 130, 9, 19, 48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
RawTBSCertificate: [48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1],
RawSubject: [48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109],
RawIssuer: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
Signature: [127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a119f65b90>{
N: {
neg: false,
abs: [3569991644534247369, 3274486715548429274, 3780411684536201245, 13386095367080826074, 4555633294449980078, 8356885564098020936, 8844059572749990990, 2540492176423473444, 5633937353057226705, 779516051327683154, 2327935533544294595, 7836233832775480006, 12443919168731326213, 5381817252368116948, 13149759042380401282, 9307346103102305840, 2490874408770263855, 8905597817602166121, 4800718759162655104, 10753677448201427049, 5736247462418790658, 11125319528290994526, 17747990166117017402, 10636684124628129003, 14904076079720377571, 4842591071448276587, 12095755243242069355, 6630948406570189761, 17118216322786437170, 12282717644381587200, 11394433495304529049, 13509030274166194453],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [9968436300229532023, 6446808008473917675, 4784458],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: ["Redmond"],
Province: ["WA"],
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "reserved.aroapp-hcp-stg.azure.com",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"WA",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Redmond",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"reserved.aroapp-hcp-stg.azure.com",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928490170, loc: nil},
NotAfter: {wall: 0, ext: 63937126570, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [1, 3, 6, 1, 4, 1, 11129, 2, 4, 2],
Critical: false,
Value: [4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 10],
Critical: false,
Value: [48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 7],
Critical: false,
Value: [48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0e\x82@*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\x82!reserved.aroapp-hcp-stg.azure.com",
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xe60\x81㠁ࠁ݆lhttp://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl\x86mhttp://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
AuthorityKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
],
DNSNames: [
"*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
"reserved.aroapp-hcp-stg.azure.com",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
"http://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
[1, 3, 6, 1, 4, 1, 311, 76, 509, 1, 1],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
{
der: [43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 7, 184, 48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
RawTBSCertificate: [48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1],
RawSubject: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
RawIssuer: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac3670>{
N: {
neg: false,
abs: [12623528037057212991, 15782976484946675180, 9131881476679932799, 16877507857597638923, 6948446969886104628, 16879364175701064122, 15349600278999781980, 9245340169127444553, 7968545109705329495, 6423711443832766155, 10594494922037898650, 17648935788986680361, 18122500389998752041, 7737602644086511993, 7258855044086458051, 5541026985182791629, 4439073283700877230, 10408130435767131132, 18089869943821258368, 11685763412349126810, 16394152652608075740, 12871239331426121127, 9800942613550528692, 18261315811323797426, 15374412250364208978, 3968124059102095807, 685806708061009512, 7838166407260346135, 16993058175899549691, 15384298262262378419, 3974146097701597185, 13694573362644585378, 13796492994165901152, 13528806942954960401, 3955259362538190321, 13074935763441801879, 2348393719768315797, 11342510827905045171, 130453718204474471, 17672562703128855053, 634844305723830904, 11538538025395239748, 5505257917472087582, 16743812431760803789, 13404336697386206403, 6770939201695035783, 3646446187118575543, 14898275314275200590, 3021489551454039149, 3930692534149328022, 11247669138812385627, 7850138742151524599, 1567520330174748017, 5626181319708709539, 16420623271324550298, 18240848139101269901, 2441811390160163564, 1395980562250961772, 3030591394459285111, 17666945104455592770, 11923213182329091847, 9136006465603430828, 13028421093649262211, 17297287522694218268],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [12560257885759602703, 4278338880794178, 3342336],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63889675383, loc: nil},
NotAfter: {wall: 0, ext: 64010808183, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 1],
Critical: false,
Value: [2, 1, 0],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 20, 2],
Critical: false,
Value: [30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65],
},
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 6, 1, 1, 255, 2, 1, 0],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xa00\x81\x9d\xa0\x81\x9a\xa0\x81\x97\x86Ihttp://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl\x86Jhttp://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true,
SubjectKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
AuthorityKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
"http://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 5, 137, 48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 1, 1, 0, 11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
RawTBSCertificate: [48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1],
RawSubject: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac37b0>{
N: {
neg: false,
abs: [8228224839953229621, 16348041302357558304, 2074854447542410873, 14334067402580427935, 12854688567428618952, 14995722382209926558, 15077658020813214080, 9553354208224609462, 902637906707506152, 15437658075602359723, 4892132923038495724, 14100513645772686328, 15699610158021110198, 116260203274558899, 7107135261377697174, 1346652938663145757, 806906029520556509, 6347563231035665119, 1416154356297248238, 13996791138010853103, 17151436068643035665, 5005973024364225782, 15423091598760460437, 11656337824329587047, 14040304327591079399, 16520357453723561614, 10318202437165161261, 17977527309428632165, 3328466382962324838, 13840171658303184680, 9417131409404627226, 5146490298140267738, 6849551933690625799, 10139571647697603240, 3552143779013315793, 15996764607220205453, 9355211167345930524, 7804203173400544275, 12256789870797553077, 5196564742993393824, 12236288164267916870, 9528338169550421684, 3418034452713204002, 17138102837348087434, 15605821747820856657, 6942198289040656341, 6027260588802198758, 4685154310243513825, 1772230202496746547, 12898472891819094090, 6838761955263670191, 15563102318165378361, 5970264090691015160, 5437520332982603245, 15757213624540617421, 6308098459188714935, 3452756883569783574, 995879279813522815, 6719559476972000178, 3606112178562803563, 6954603544383871929, 16027092704045215034, 8191838635022848599, 16134862131230222352],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [5148676810827874504, 796129072041760688],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63883382400, loc: nil},
NotAfter: {wall: 0, ext: 64012204799, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0907\xa05\xa03\x861http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
AuthorityKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
OCSPServer: [
"http://ocsp.digicert.com",
],
IssuingCertificateURL: [
"http://cacerts.digicert.com/DigiCertGlobalRootG2.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 3, 142, 48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
RawTBSCertificate: [48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1],
RawSubject: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a11782c840>{
N: {
neg: false,
abs: [13518897546323503493, 12614377001919816122, 8557832239203824717, 7433462159093546310, 9504020990483827396, 702024576034523901, 5333964605457796894, 8971906913408307990, 18252057450268743444, 5039799191733612724, 10346373305228545133, 18107087016177239977, 3285062882728703971, 2137966295034491701, 13110919595711778188, 17490549066068800107, 17827376715711028367, 6958731023901346072, 11071174057932843776, 11902983757762476756, 4850057369827327278, 16198203776063958047, 2241017827087547863, 13230558770443115738, 9142400105587256400, 4952379155939763112, 7768617195409377572, 9912770753372816922, 14554377099045433914, 13412012197143714132, 12855684210282069830, 13490476835732679625],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [13486139796020132581, 232764303710726560],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63510955200, loc: nil},
NotAfter: {wall: 0, ext: 64282766400, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: nil,
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
AuthorityKeyId: nil,
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x36a1182f5208 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:525]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x36a1186625c0>:
VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*fmt.wrapError | 0x36a1186623e0>{
msg: "ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x36a118d74840>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
Err: <*tls.CertificateVerificationError | 0x36a118d74810>{
UnverifiedCertificates: [
{
Raw: [48, 130, 9, 19, 48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
RawTBSCertificate: [48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1],
RawSubject: [48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109],
RawIssuer: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
Signature: [127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a119f65b90>{
N: {
neg: false,
abs: [3569991644534247369, 3274486715548429274, 3780411684536201245, 13386095367080826074, 4555633294449980078, 8356885564098020936, 8844059572749990990, 2540492176423473444, 5633937353057226705, 779516051327683154, 2327935533544294595, 7836233832775480006, 12443919168731326213, 5381817252368116948, 13149759042380401282, 9307346103102305840, 2490874408770263855, 8905597817602166121, 4800718759162655104, 10753677448201427049, 5736247462418790658, 11125319528290994526, 17747990166117017402, 10636684124628129003, 14904076079720377571, 4842591071448276587, 12095755243242069355, 6630948406570189761, 17118216322786437170, 12282717644381587200, 11394433495304529049, 13509030274166194453],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [9968436300229532023, 6446808008473917675, 4784458],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: ["Redmond"],
Province: ["WA"],
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "reserved.aroapp-hcp-stg.azure.com",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"WA",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Redmond",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"reserved.aroapp-hcp-stg.azure.com",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928490170, loc: nil},
NotAfter: {wall: 0, ext: 63937126570, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [1, 3, 6, 1, 4, 1, 11129, 2, 4, 2],
Critical: false,
Value: [4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 10],
Critical: false,
Value: [48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 7],
Critical: false,
Value: [48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0e\x82@*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\x82!reserved.aroapp-hcp-stg.azure.com",
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xe60\x81㠁ࠁ݆lhttp://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl\x86mhttp://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
AuthorityKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
],
DNSNames: [
"*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
"reserved.aroapp-hcp-stg.azure.com",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
"http://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
[1, 3, 6, 1, 4, 1, 311, 76, 509, 1, 1],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
{
der: [43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 7, 184, 48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
RawTBSCertificate: [48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1],
RawSubject: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
RawIssuer: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac3670>{
N: {
neg: false,
abs: [12623528037057212991, 15782976484946675180, 9131881476679932799, 16877507857597638923, 6948446969886104628, 16879364175701064122, 15349600278999781980, 9245340169127444553, 7968545109705329495, 6423711443832766155, 10594494922037898650, 17648935788986680361, 18122500389998752041, 7737602644086511993, 7258855044086458051, 5541026985182791629, 4439073283700877230, 10408130435767131132, 18089869943821258368, 11685763412349126810, 16394152652608075740, 12871239331426121127, 9800942613550528692, 18261315811323797426, 15374412250364208978, 3968124059102095807, 685806708061009512, 7838166407260346135, 16993058175899549691, 15384298262262378419, 3974146097701597185, 13694573362644585378, 13796492994165901152, 13528806942954960401, 3955259362538190321, 13074935763441801879, 2348393719768315797, 11342510827905045171, 130453718204474471, 17672562703128855053, 634844305723830904, 11538538025395239748, 5505257917472087582, 16743812431760803789, 13404336697386206403, 6770939201695035783, 3646446187118575543, 14898275314275200590, 3021489551454039149, 3930692534149328022, 11247669138812385627, 7850138742151524599, 1567520330174748017, 5626181319708709539, 16420623271324550298, 18240848139101269901, 2441811390160163564, 1395980562250961772, 3030591394459285111, 17666945104455592770, 11923213182329091847, 9136006465603430828, 13028421093649262211, 17297287522694218268],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [12560257885759602703, 4278338880794178, 3342336],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63889675383, loc: nil},
NotAfter: {wall: 0, ext: 64010808183, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 1],
Critical: false,
Value: [2, 1, 0],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 20, 2],
Critical: false,
Value: [30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65],
},
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 6, 1, 1, 255, 2, 1, 0],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xa00\x81\x9d\xa0\x81\x9a\xa0\x81\x97\x86Ihttp://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl\x86Jhttp://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true,
SubjectKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
AuthorityKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
"http://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 5, 137, 48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 1, 1, 0, 11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
RawTBSCertificate: [48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1],
RawSubject: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac37b0>{
N: {
neg: false,
abs: [8228224839953229621, 16348041302357558304, 2074854447542410873, 14334067402580427935, 12854688567428618952, 14995722382209926558, 15077658020813214080, 9553354208224609462, 902637906707506152, 15437658075602359723, 4892132923038495724, 14100513645772686328, 15699610158021110198, 116260203274558899, 7107135261377697174, 1346652938663145757, 806906029520556509, 6347563231035665119, 1416154356297248238, 13996791138010853103, 17151436068643035665, 5005973024364225782, 15423091598760460437, 11656337824329587047, 14040304327591079399, 16520357453723561614, 10318202437165161261, 17977527309428632165, 3328466382962324838, 13840171658303184680, 9417131409404627226, 5146490298140267738, 6849551933690625799, 10139571647697603240, 3552143779013315793, 15996764607220205453, 9355211167345930524, 7804203173400544275, 12256789870797553077, 5196564742993393824, 12236288164267916870, 9528338169550421684, 3418034452713204002, 17138102837348087434, 15605821747820856657, 6942198289040656341, 6027260588802198758, 4685154310243513825, 1772230202496746547, 12898472891819094090, 6838761955263670191, 15563102318165378361, 5970264090691015160, 5437520332982603245, 15757213624540617421, 6308098459188714935, 3452756883569783574, 995879279813522815, 6719559476972000178, 3606112178562803563, 6954603544383871929, 16027092704045215034, 8191838635022848599, 16134862131230222352],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [5148676810827874504, 796129072041760688],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63883382400, loc: nil},
NotAfter: {wall: 0, ext: 64012204799, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0907\xa05\xa03\x861http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
AuthorityKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
OCSPServer: [
"http://ocsp.digicert.com",
],
IssuingCertificateURL: [
"http://cacerts.digicert.com/DigiCertGlobalRootG2.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 3, 142, 48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
RawTBSCertificate: [48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1],
RawSubject: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a11782c840>{
N: {
neg: false,
abs: [13518897546323503493, 12614377001919816122, 8557832239203824717, 7433462159093546310, 9504020990483827396, 702024576034523901, 5333964605457796894, 8971906913408307990, 18252057450268743444, 5039799191733612724, 10346373305228545133, 18107087016177239977, 3285062882728703971, 2137966295034491701, 13110919595711778188, 17490549066068800107, 17827376715711028367, 6958731023901346072, 11071174057932843776, 11902983757762476756, 4850057369827327278, 16198203776063958047, 2241017827087547863, 13230558770443115738, 9142400105587256400, 4952379155939763112, 7768617195409377572, 9912770753372816922, 14554377099045433914, 13412012197143714132, 12855684210282069830, 13490476835732679625],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [13486139796020132581, 232764303710726560],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63510955200, loc: nil},
NotAfter: {wall: 0, ext: 64282766400, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: nil,
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
AuthorityKeyId: nil,
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x36a1182f5208 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-<id>])full failure pattern: expected exactly 1 VM(s) for node pool <nodepool>, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-<id>]) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 1 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:886]: Timed out after 900.004s.
node pool np-1 never detached the day 2 MI /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/acr-pull-2pg5nqg2c8wm/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi-2 after removal
Expected success, but got an error:
<*errors.errorString | 0x18a468a961b0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:886]: Timed out after 900.004s.
node pool np-1 never detached the day 2 MI /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/acr-pull-2pg5nqg2c8wm/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi-2 after removal
Expected success, but got an error:
<*errors.errorString | 0x18a468a961b0>:
expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])
{
s: "expected exactly 1 VM(s) for node pool np-1, got 2 ([cluster-np-1-6qhfh-vkcdv cluster-np-1-l96tz-mplj4])",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 50.00% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 50.00% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 58.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 58.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 58.33% | 12 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 7 | 9.09%7 of 77 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 19 · Oct 29: 3 · Oct 30: 0 | DEV, INT, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x1def056ca980>:
failed to create HCP cluster ea-listzz8ctk: failed waiting for cluster="ea-listzz8ctk" in resourcegroup="ea-list-4jljqltdpf22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51",
"name": "671f379d-8472-4150-82a8-d71efe3bef51",
"status": "Failed",
"startTime": "2026-10-29T22:40:05.142063562Z",
"endTime": "2026-10-29T23:09:09.015450129Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
correlation ID: 462ab43f-575c-42b1-ab0d-20c41a4d0046
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x1def056ca980>:
failed to create HCP cluster ea-listzz8ctk: failed waiting for cluster="ea-listzz8ctk" in resourcegroup="ea-list-4jljqltdpf22" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51",
"name": "671f379d-8472-4150-82a8-d71efe3bef51",
"status": "Failed",
"startTime": "2026-10-29T22:40:05.142063562Z",
"endTime": "2026-10-29T23:09:09.015450129Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/671f379d-8472-4150-82a8-d71efe3bef51
correlation ID: 462ab43f-575c-42b1-ab0d-20c41a4d0046
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x241c81f32b80>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-58txd54zlccz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"name": "c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"status": "Failed",
"startTime": "2026-10-29T14:34:25.586865421Z",
"endTime": "2026-10-29T15:03:34.559330088Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
correlation ID: 5c469ba9-6c92-4dd6-9ba5-04a68fee9ef5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x241c81f32b80>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-58txd54zlccz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"name": "c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7",
"status": "Failed",
"startTime": "2026-10-29T14:34:25.586865421Z",
"endTime": "2026-10-29T15:03:34.559330088Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/c6f1a5b2-75ae-48b0-a26b-e94834d0ddc7
correlation ID: 5c469ba9-6c92-4dd6-9ba5-04a68fee9ef5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fw4jcb" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x5b710c1e4e0>:
failed to create HCP cluster cp-ystream-upgrade-4-23-fw4jcb: failed waiting for cluster="cp-ystream-upgrade-4-23-fw4jcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-sftgrsslrwdw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c",
"name": "b53af301-d50b-4096-b43d-a7144a6a296c",
"status": "Failed",
"startTime": "2026-10-29T06:25:50.628347504Z",
"endTime": "2026-10-29T06:54:52.041701794Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
correlation ID: 4b2677ce-b17f-435d-9634-3583d8f686b3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-fw4jcb" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x5b710c1e4e0>:
failed to create HCP cluster cp-ystream-upgrade-4-23-fw4jcb: failed waiting for cluster="cp-ystream-upgrade-4-23-fw4jcb" in resourcegroup="rg-cp-ystream-upgrade-4-23-sftgrsslrwdw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c",
"name": "b53af301-d50b-4096-b43d-a7144a6a296c",
"status": "Failed",
"startTime": "2026-10-29T06:25:50.628347504Z",
"endTime": "2026-10-29T06:54:52.041701794Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/b53af301-d50b-4096-b43d-a7144a6a296c
correlation ID: 4b2677ce-b17f-435d-9634-3583d8f686b3
client request ID: unavailable
occurredContributing tests (17)
Affected runs (7)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 3 | 3.90%3 of 77 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 23: 10 · Oct 24: 5 · Oct 25: 0 · Oct 26: 5 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (11)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 2 | 2.60%2 of 77 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 23: 1 · Oct 24: 2 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x5c10dd80030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5c10d35ddc0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5c10d35dda0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5c10db35540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5c10dd88020>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5c10dd88000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5c10dbae320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x5c10dd80030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5c10d35ddc0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5c10d35dda0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5c10db35540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5c10dd88020>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5c10dd88000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5c10dbae320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-67pgbb" is viable
Unexpected error:
<*errors.joinError | 0x21cecb2413f8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21cecb85e9a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21cecb85e980>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21cecae4e500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21cecb471000>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21cecb470fe0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21cecb1806e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-67pgbb" is viable
Unexpected error:
<*errors.joinError | 0x21cecb2413f8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21cecb85e9a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21cecb85e980>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21cecae4e500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21cecb471000>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21cecb470fe0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21cecb1806e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x13bc429c0870>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x13bc42aa9920>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x13bc42aa9900>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x13bc41a5d400>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x13bc425c16c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x13bc425c16a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x13bc416d8960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x13bc429c0870>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x13bc42aa9920>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x13bc42aa9900>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x13bc41a5d400>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x13bc425c16c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x13bc425c16a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x13bc416d8960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment | e2e | 1 | 1.30%1 of 77 job runs affected | NoiseSignal: Noise — struct/object fragment | Show trend detailsOct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 5 | 6.49%5 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 2 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2535834a99c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rbmj59bntg5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1",
"name": "49137998-c82d-463c-9cd4-a8775024ebc1",
"status": "Failed",
"startTime": "2026-10-29T04:26:18.62948754Z",
"endTime": "2026-10-29T04:45:22.348417217Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
correlation ID: 40719e78-c8c1-4a49-a0a8-7e5f3f0cfa60
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2535834a99c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rbmj59bntg5w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1",
"name": "49137998-c82d-463c-9cd4-a8775024ebc1",
"status": "Failed",
"startTime": "2026-10-29T04:26:18.62948754Z",
"endTime": "2026-10-29T04:45:22.348417217Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/49137998-c82d-463c-9cd4-a8775024ebc1
correlation ID: 40719e78-c8c1-4a49-a0a8-7e5f3f0cfa60
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0xa04a5fdf140>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ffbhgrn8qbrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747",
"name": "ade1a10f-df33-4040-9e11-b9632ef21747",
"status": "Failed",
"startTime": "2026-10-29T01:55:59.309285263Z",
"endTime": "2026-10-29T02:15:01.505600334Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
correlation ID: 0ca07d4a-0c93-4a4d-806f-591a9bde4d58
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0xa04a5fdf140>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ffbhgrn8qbrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747",
"name": "ade1a10f-df33-4040-9e11-b9632ef21747",
"status": "Failed",
"startTime": "2026-10-29T01:55:59.309285263Z",
"endTime": "2026-10-29T02:15:01.505600334Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/ade1a10f-df33-4040-9e11-b9632ef21747
correlation ID: 0ca07d4a-0c93-4a4d-806f-591a9bde4d58
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 3 | 3.90%3 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 23: 7 · Oct 24: 5 · Oct 25: 0 · Oct 26: 4 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (9)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 16 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x22868c3a2660>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"status": "Failed",
"startTime": "2026-10-28T19:24:09.588021132Z",
"endTime": "2026-10-28T19:53:13.758028769Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x22868c3a2660>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"status": "Failed",
"startTime": "2026-10-28T19:24:09.588021132Z",
"endTime": "2026-10-28T19:53:13.758028769Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x263f7a1ecb40>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
"name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
"status": "Failed",
"startTime": "2026-10-28T20:14:34.589492538Z",
"endTime": "2026-10-28T20:43:34.88292258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x263f7a1ecb40>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
"name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
"status": "Failed",
"startTime": "2026-10-28T20:14:34.589492538Z",
"endTime": "2026-10-28T20:43:34.88292258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
<*fmt.wrapError | 0x2d29037be7e0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
"name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
"status": "Failed",
"startTime": "2026-10-28T20:12:38.563887566Z",
"endTime": "2026-10-28T20:41:44.890635353Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
<*fmt.wrapError | 0x2d29037be7e0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
"name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
"status": "Failed",
"startTime": "2026-10-28T20:12:38.563887566Z",
"endTime": "2026-10-28T20:41:44.890635353Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
client request ID: unavailable
occurredContributing tests (13)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 14 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
<*fmt.wrapError | 0x15a8256fc4a0>:
failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"status": "Failed",
"startTime": "2026-10-28T20:04:15.708912895Z",
"endTime": "2026-10-28T20:33:16.216619801Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
<*fmt.wrapError | 0x15a8256fc4a0>:
failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"status": "Failed",
"startTime": "2026-10-28T20:04:15.708912895Z",
"endTime": "2026-10-28T20:33:16.216619801Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xd149dec0ae0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.936404311Z",
"endTime": "2026-10-28T19:53:23.678300779Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xd149dec0ae0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.936404311Z",
"endTime": "2026-10-28T19:53:23.678300779Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x3b3d00e688d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"status": "Failed",
"startTime": "2026-10-28T20:59:31.361013478Z",
"endTime": "2026-10-28T21:28:40.621642045Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x3b3d00e688d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"status": "Failed",
"startTime": "2026-10-28T20:59:31.361013478Z",
"endTime": "2026-10-28T21:28:40.621642045Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
client request ID: unavailable
occurredContributing tests (11)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 12 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
<*fmt.wrapError | 0xba6b52aa80>:
failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"status": "Failed",
"startTime": "2026-10-28T20:10:20.02531103Z",
"endTime": "2026-10-28T20:39:25.022579343Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
correlation ID: 15324628-570b-42b9-80a7-dbd435939211
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
<*fmt.wrapError | 0xba6b52aa80>:
failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"status": "Failed",
"startTime": "2026-10-28T20:10:20.02531103Z",
"endTime": "2026-10-28T20:39:25.022579343Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
correlation ID: 15324628-570b-42b9-80a7-dbd435939211
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
<*fmt.wrapError | 0x104ef3fa7e0>:
failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
"name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
"status": "Failed",
"startTime": "2026-10-28T20:10:48.049950576Z",
"endTime": "2026-10-28T20:39:55.034999436Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
<*fmt.wrapError | 0x104ef3fa7e0>:
failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
"name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
"status": "Failed",
"startTime": "2026-10-28T20:10:48.049950576Z",
"endTime": "2026-10-28T20:39:55.034999436Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x1da2d78427a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"status": "Failed",
"startTime": "2026-10-28T19:26:05.081696809Z",
"endTime": "2026-10-28T19:55:13.864538233Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x1da2d78427a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"status": "Failed",
"startTime": "2026-10-28T19:26:05.081696809Z",
"endTime": "2026-10-28T19:55:13.864538233Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
client request ID: unavailable
occurredContributing tests (9)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x34cbd3d68900>:
failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
"name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
"status": "Failed",
"startTime": "2026-10-28T20:14:32.731497539Z",
"endTime": "2026-10-28T20:43:34.905565303Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x34cbd3d68900>:
failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
"name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
"status": "Failed",
"startTime": "2026-10-28T20:14:32.731497539Z",
"endTime": "2026-10-28T20:43:34.905565303Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x118c82e68960>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"status": "Failed",
"startTime": "2026-10-28T20:58:03.268943167Z",
"endTime": "2026-10-28T21:27:10.630246232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x118c82e68960>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"status": "Failed",
"startTime": "2026-10-28T20:58:03.268943167Z",
"endTime": "2026-10-28T21:27:10.630246232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
<*fmt.wrapError | 0x3e94b832e820>:
failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
"name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
"status": "Failed",
"startTime": "2026-10-28T20:06:06.856819198Z",
"endTime": "2026-10-28T20:35:15.000751357Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
<*fmt.wrapError | 0x3e94b832e820>:
failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
"name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
"status": "Failed",
"startTime": "2026-10-28T20:06:06.856819198Z",
"endTime": "2026-10-28T20:35:15.000751357Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x1d67385a4720>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"status": "Failed",
"startTime": "2026-10-28T19:24:18.787179838Z",
"endTime": "2026-10-28T19:53:23.770490111Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x1d67385a4720>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"status": "Failed",
"startTime": "2026-10-28T19:24:18.787179838Z",
"endTime": "2026-10-28T19:53:23.770490111Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x23a60cd6e680>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"status": "Failed",
"startTime": "2026-10-28T19:24:25.094565507Z",
"endTime": "2026-10-28T19:53:33.692195544Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x23a60cd6e680>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"status": "Failed",
"startTime": "2026-10-28T19:24:25.094565507Z",
"endTime": "2026-10-28T19:53:33.692195544Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x34142edd6a40>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
"name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
"status": "Failed",
"startTime": "2026-10-28T19:24:14.723008935Z",
"endTime": "2026-10-28T19:53:23.809094912Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x34142edd6a40>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
"name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
"status": "Failed",
"startTime": "2026-10-28T19:24:14.723008935Z",
"endTime": "2026-10-28T19:53:23.809094912Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x2a9a9c302780>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"status": "Failed",
"startTime": "2026-10-28T20:14:46.976794807Z",
"endTime": "2026-10-28T20:43:55.053369978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x2a9a9c302780>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"status": "Failed",
"startTime": "2026-10-28T20:14:46.976794807Z",
"endTime": "2026-10-28T20:43:55.053369978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0xe23c97da540>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"status": "Failed",
"startTime": "2026-10-28T20:28:51.316103778Z",
"endTime": "2026-10-28T20:57:55.004654509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0xe23c97da540>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"status": "Failed",
"startTime": "2026-10-28T20:28:51.316103778Z",
"endTime": "2026-10-28T20:57:55.004654509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x38a796b1a740>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
"name": "499291aa-b279-429d-80ba-7786fc6cfa93",
"status": "Failed",
"startTime": "2026-10-28T20:38:17.873619905Z",
"endTime": "2026-10-28T21:07:24.99192406Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x38a796b1a740>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
"name": "499291aa-b279-429d-80ba-7786fc6cfa93",
"status": "Failed",
"startTime": "2026-10-28T20:38:17.873619905Z",
"endTime": "2026-10-28T21:07:24.99192406Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1c7b6c0a8440>:
failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
"name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
"status": "Failed",
"startTime": "2026-10-28T20:56:27.61784978Z",
"endTime": "2026-10-28T21:25:30.606619249Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1c7b6c0a8440>:
failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
"name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
"status": "Failed",
"startTime": "2026-10-28T20:56:27.61784978Z",
"endTime": "2026-10-28T21:25:30.606619249Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0x2c03b21d6c20>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"status": "Failed",
"startTime": "2026-10-28T20:14:37.658468995Z",
"endTime": "2026-10-28T20:43:44.89685741Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0x2c03b21d6c20>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"status": "Failed",
"startTime": "2026-10-28T20:14:37.658468995Z",
"endTime": "2026-10-28T20:43:44.89685741Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurredContributing tests (4)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x14a5e2008300>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
"name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
"status": "Failed",
"startTime": "2026-10-28T19:26:12.743400806Z",
"endTime": "2026-10-28T19:55:13.7057976Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x14a5e2008300>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
"name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
"status": "Failed",
"startTime": "2026-10-28T19:26:12.743400806Z",
"endTime": "2026-10-28T19:55:13.7057976Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x1683f5097100>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"status": "Failed",
"startTime": "2026-10-28T19:28:18.023653922Z",
"endTime": "2026-10-28T19:57:23.841118908Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x1683f5097100>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"status": "Failed",
"startTime": "2026-10-28T19:28:18.023653922Z",
"endTime": "2026-10-28T19:57:23.841118908Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x13016dc5e5e0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.851456324Z",
"endTime": "2026-10-28T20:41:34.873934441Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x13016dc5e5e0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.851456324Z",
"endTime": "2026-10-28T20:41:34.873934441Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xfa629f24ea0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xfa629c70d20>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xfa629f24ea0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xfa629c70d20>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 1 · Oct 30: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x243ce2483b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-r4jthp6b8c2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67",
"name": "4f6eb8f6-434d-4041-9e38-be33ab685c67",
"status": "Failed",
"startTime": "2026-10-29T13:48:20.576616469Z",
"endTime": "2026-10-29T14:07:20.745392823Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
correlation ID: 72c0068c-b7b6-49ea-9082-ad530d20a571
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x243ce2483b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-r4jthp6b8c2n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67",
"name": "4f6eb8f6-434d-4041-9e38-be33ab685c67",
"status": "Failed",
"startTime": "2026-10-29T13:48:20.576616469Z",
"endTime": "2026-10-29T14:07:20.745392823Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-9lkz9-h7ldx: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/4f6eb8f6-434d-4041-9e38-be33ab685c67
correlation ID: 72c0068c-b7b6-49ea-9082-ad530d20a571
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x26d436e0e5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rgpk9lmtvnvv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105",
"name": "357cbb72-4fad-4dd1-860d-138b89ef8105",
"status": "Failed",
"startTime": "2026-10-28T23:13:44.545799939Z",
"endTime": "2026-10-28T23:32:49.563451274Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pkf64-8lv9b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pkf64-8lv9b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
correlation ID: 8689aa1d-b711-4817-af27-601b622fb10c
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x26d436e0e5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rgpk9lmtvnvv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105",
"name": "357cbb72-4fad-4dd1-860d-138b89ef8105",
"status": "Failed",
"startTime": "2026-10-28T23:13:44.545799939Z",
"endTime": "2026-10-28T23:32:49.563451274Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pkf64-8lv9b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pkf64-8lv9b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-rgpk9lmtvnvv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pkf64-8lv9b to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/357cbb72-4fad-4dd1-860d-138b89ef8105
correlation ID: 8689aa1d-b711-4817-af27-601b622fb10c
client request ID: unavailableContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed Contributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.60%2 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 5 · Oct 24: 0 · Oct 25: 7 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-5gtk26"
Unexpected error:
<*fmt.wrapError | 0x187dce020660>:
DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x187dce1941c0>{
msg: "DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-5gtk26"
Unexpected error:
<*fmt.wrapError | 0x187dce020660>:
DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x187dce1941c0>{
msg: "DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33aec8dca6e0>:
DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33aec9a24040>{
msg: "DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33aec8dca6e0>:
DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33aec9a24040>{
msg: "DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xaa72ff3eb40>:
DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xaa72fd7c9e0>{
msg: "DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xaa72ff3eb40>:
DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xaa72fd7c9e0>{
msg: "DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (7)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0xc983e72b20>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
"name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.13733405Z",
"endTime": "2026-10-28T19:53:23.880336461Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0xc983e72b20>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
"name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.13733405Z",
"endTime": "2026-10-28T19:53:23.880336461Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3981bcd4a5c0>:
failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
"name": "fa758444-9da4-4261-b65a-fb9af0416aae",
"status": "Failed",
"startTime": "2026-10-28T20:11:05.048147318Z",
"endTime": "2026-10-28T20:40:15.036753709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3981bcd4a5c0>:
failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
"name": "fa758444-9da4-4261-b65a-fb9af0416aae",
"status": "Failed",
"startTime": "2026-10-28T20:11:05.048147318Z",
"endTime": "2026-10-28T20:40:15.036753709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x9790da22600>:
failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
"name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
"status": "Failed",
"startTime": "2026-10-28T20:55:25.009646036Z",
"endTime": "2026-10-28T21:24:30.625209621Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x9790da22600>:
failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
"name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
"status": "Failed",
"startTime": "2026-10-28T20:55:25.009646036Z",
"endTime": "2026-10-28T21:24:30.625209621Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
client request ID: unavailable
occurredContributing tests (6)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x16967f5def20>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
"name": "54478daa-47e5-4530-91c2-342cc29a1749",
"status": "Failed",
"startTime": "2026-10-28T19:24:21.628677591Z",
"endTime": "2026-10-28T19:53:23.838328082Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x16967f5def20>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
"name": "54478daa-47e5-4530-91c2-342cc29a1749",
"status": "Failed",
"startTime": "2026-10-28T19:24:21.628677591Z",
"endTime": "2026-10-28T19:53:23.838328082Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d00e53ba4e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"status": "Failed",
"startTime": "2026-10-28T19:24:17.131948207Z",
"endTime": "2026-10-28T19:53:23.815769706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d00e53ba4e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"status": "Failed",
"startTime": "2026-10-28T19:24:17.131948207Z",
"endTime": "2026-10-28T19:53:23.815769706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x1c6ab1a4a4c0>:
failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
"name": "13f26b26-2324-48a4-82df-d8b324b13996",
"status": "Failed",
"startTime": "2026-10-28T20:52:21.783931077Z",
"endTime": "2026-10-28T21:21:30.723052246Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x1c6ab1a4a4c0>:
failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
"name": "13f26b26-2324-48a4-82df-d8b324b13996",
"status": "Failed",
"startTime": "2026-10-28T20:52:21.783931077Z",
"endTime": "2026-10-28T21:21:30.723052246Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 3 · Oct 24: 0 · Oct 25: 2 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x33a1f1315158>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x33a1f0cea780>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f0bd2f00>{
msg: "DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f13e4360>{
msg: "DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x33a1f1315158>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x33a1f0cea780>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f0bd2f00>{
msg: "DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f13e4360>{
msg: "DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xeeb7f5e0a68>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xeeb7f3b69e0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f5137a0>{
msg: "DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f512040>{
msg: "DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xeeb7f5e0a68>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xeeb7f3b69e0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f5137a0>{
msg: "DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f512040>{
msg: "DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x2df91d94ebc0>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
"name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
"status": "Failed",
"startTime": "2026-10-28T19:52:14.429975606Z",
"endTime": "2026-10-28T20:21:16.149275515Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x2df91d94ebc0>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
"name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
"status": "Failed",
"startTime": "2026-10-28T19:52:14.429975606Z",
"endTime": "2026-10-28T20:21:16.149275515Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2874318ca1e0>:
resource group "rg-nightly-4-21-kv7pzz4ch49v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-21-7x886k" in resourcegroup="rg-nightly-4-21-kv7pzz4ch49v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"name": "35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"status": "Failed",
"startTime": "2026-10-24T23:51:18.597955269Z",
"endTime": "2026-10-25T00:15:22.050322008Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
correlation ID: e4ebdf9b-312f-4c03-affc-b5c4cbd39c75
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2874318ca1e0>:
resource group "rg-nightly-4-21-kv7pzz4ch49v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-21-7x886k" in resourcegroup="rg-nightly-4-21-kv7pzz4ch49v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"name": "35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"status": "Failed",
"startTime": "2026-10-24T23:51:18.597955269Z",
"endTime": "2026-10-25T00:15:22.050322008Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
correlation ID: e4ebdf9b-312f-4c03-affc-b5c4cbd39c75
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...full failure pattern: cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 23: 1 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x471790ed8f0>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x4717a706200>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x471790ed8f0>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x4717a706200>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-<id>full failure pattern: DNS for route host agnhost-e2e-sample-app-<id> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | DEV, INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x1d8dba328160>:
DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1d8dba3280e0>{
msg: "DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x1d8dba328160>:
DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1d8dba3280e0>{
msg: "DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x219ad233aca0>:
failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
"name": "e768a2f1-bf75-4afe-9acf-741914b73091",
"status": "Failed",
"startTime": "2026-10-28T20:11:31.970768227Z",
"endTime": "2026-10-28T20:40:35.054714998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x219ad233aca0>:
failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
"name": "e768a2f1-bf75-4afe-9acf-741914b73091",
"status": "Failed",
"startTime": "2026-10-28T20:11:31.970768227Z",
"endTime": "2026-10-28T20:40:35.054714998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, ignition-server-proxy, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1c9f11d305c0>:
failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
"name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
"status": "Failed",
"startTime": "2026-10-28T20:57:10.002650956Z",
"endTime": "2026-10-28T21:26:10.63929862Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1c9f11d305c0>:
failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
"name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
"status": "Failed",
"startTime": "2026-10-28T20:57:10.002650956Z",
"endTime": "2026-10-28T21:26:10.63929862Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-controller-manager, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x3da3f19209c0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"status": "Failed",
"startTime": "2026-10-28T20:14:36.875741431Z",
"endTime": "2026-10-28T20:43:44.887394978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
correlation ID: 84abeaa9-4051-418b-b066-32246401955a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x3da3f19209c0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"status": "Failed",
"startTime": "2026-10-28T20:14:36.875741431Z",
"endTime": "2026-10-28T20:43:44.887394978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
correlation ID: 84abeaa9-4051-418b-b066-32246401955a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x2840a3490800>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
"name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
"status": "Failed",
"startTime": "2026-10-28T20:48:29.562853635Z",
"endTime": "2026-10-28T21:17:30.601011159Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x2840a3490800>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
"name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
"status": "Failed",
"startTime": "2026-10-28T20:48:29.562853635Z",
"endTime": "2026-10-28T21:17:30.601011159Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x3290e061a680>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"status": "Failed",
"startTime": "2026-10-28T20:57:27.436526224Z",
"endTime": "2026-10-28T21:26:30.622804028Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x3290e061a680>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"status": "Failed",
"startTime": "2026-10-28T20:57:27.436526224Z",
"endTime": "2026-10-28T21:26:30.622804028Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x1d947782ea20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
"name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
"status": "Failed",
"startTime": "2026-10-28T20:01:08.393167298Z",
"endTime": "2026-10-28T20:30:16.146752142Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x1d947782ea20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
"name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
"status": "Failed",
"startTime": "2026-10-28T20:01:08.393167298Z",
"endTime": "2026-10-28T20:30:16.146752142Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x28ad82ebac20>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
"name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
"status": "Failed",
"startTime": "2026-10-28T20:01:11.67931549Z",
"endTime": "2026-10-28T20:30:16.178438369Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x28ad82ebac20>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
"name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
"status": "Failed",
"startTime": "2026-10-28T20:01:11.67931549Z",
"endTime": "2026-10-28T20:30:16.178438369Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, kube-controller-manager, kube-scheduler]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x1308b97a65a0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"status": "Failed",
"startTime": "2026-10-28T20:58:09.127282806Z",
"endTime": "2026-10-28T21:27:10.682711777Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
correlation ID: 49b88924-064f-4f52-9480-109cf607a390
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x1308b97a65a0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"status": "Failed",
"startTime": "2026-10-28T20:58:09.127282806Z",
"endTime": "2026-10-28T21:27:10.682711777Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
correlation ID: 49b88924-064f-4f52-9480-109cf607a390
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
<*fmt.wrapError | 0xd7eab0d0620>:
failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
"name": "36728157-067b-4375-a959-9c9ce43e534b",
"status": "Failed",
"startTime": "2026-10-28T20:56:26.269135944Z",
"endTime": "2026-10-28T21:25:30.687402107Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
<*fmt.wrapError | 0xd7eab0d0620>:
failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
"name": "36728157-067b-4375-a959-9c9ce43e534b",
"status": "Failed",
"startTime": "2026-10-28T20:56:26.269135944Z",
"endTime": "2026-10-28T21:25:30.687402107Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: interfaces creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such hostfull failure pattern: public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.30%1 of 77 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x1bbed2a5bb90>:
Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x1bbed27f1680>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x1bbed27f1450>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x1bbed2a5bb90>:
Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x1bbed27f1680>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x1bbed27f1450>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||