Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 208 | 34.62% (off track) | -60.38pp | +5.36pp | 90.59% (183/202) | -4.22pp | 39.34% (72/183) | +5.75pp |
| INT | 95% - All E2E job runs | 0 | n/a (insufficient data) | n/a | n/a | n/a | n/a | n/a | n/a |
| STG | 95% - All E2E job runs | 12 | 16.67% (off track) | -78.33pp | -1.85pp | n/a | n/a | 16.67% (2/12) | -7.14pp |
| PROD | 95% - All E2E job runs | 37 | 43.24% (off track) | -51.76pp | +15.47pp | n/a | n/a | 45.71% (16/35) | +14.46pp |
Environment: DEV
E2E Jobs
491
Success Rate
25.87%
Provision success
Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.89.29% (417/467)
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.30.46% (127/417)
E2E Jobs (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the number of runs in that DEV-only goal basis.208
Success Rate (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. This card shows the overall success rate for that DEV-only goal basis.34.62%
Provision success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Provisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step.90.59% (183/202)
E2E success (post-good + batches)
These PR-regression-filtered DEV metrics include runs on the final commit of a PR that later merged, plus Tide batch retests of PRs that passed their individual e2e-parallel checks. Each run counts once. Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.39.34% (72/183)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
Successful runs (post-good + batches)
Provision failures
E2E failures
Other failures
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key "globalMSIId" not foundfull failure pattern: failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key "globalMSIId" not found Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | provision | 6 | 1.22%6 of 491 job runs affected | RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 4 · Oct 17: 0 · Oct 18: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-18T15:38:05.575Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-10-18T15:38:06.914Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found" time=2026-10-18T12:10:56.819Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-10-18T12:10:57.618Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found" time=2026-10-16T23:16:01.424Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-10-16T23:16:02.804Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to get input values: variable invalid: failed to get value for input output.globalMSIId: key \"globalMSIId\" not found" Contributing tests (1)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 46 | 9.37%46 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 11: 2 · Oct 12: 8 · Oct 13: 9 · Oct 14: 12 · Oct 15: 11 · Oct 16: 6 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-16T17:47:40Z Ended: 2026-10-16T17:58:40Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1762432-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-16T17:00:31Z Ended: 2026-10-16T17:16:34Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5648768-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-10-16T17:00:31Z Ended: 2026-10-16T17:14:30Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5648768-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-16T14:54:18Z Ended: 2026-10-16T15:01:19Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3268992-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (46) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 8 days | alert | 37 | 7.54%37 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 8 days | Show trend detailsOct 11: 4 · Oct 12: 5 · Oct 13: 6 · Oct 14: 4 · Oct 15: 10 · Oct 16: 9 · Oct 17: 1 · Oct 18: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 38 time(s) Firing 1: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-tmw4b", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-tmw4b has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-4fdv6", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-4fdv6 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-5vc4f", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-5vc4f has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-dfx48", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-dfx48 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-tmkfh", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-tmkfh has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-ldq7g", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-ldq7g has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-4prhm", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-4prhm has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-mtvd8", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-mtvd8 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-w9lfx", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-w9lfx has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-m6kzd", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-m6kzd has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-gf4mg", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-gf4mg has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-jtg79", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-jtg79 has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-gbc9d", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-gbc9d has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:03Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-vnwdn", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-vnwdn has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-cvdcq", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-cvdcq has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-jp4zq", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-jp4zq has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-lcvwv", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-lcvwv has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-nllhh", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-nllhh has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-x8lvp", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-x8lvp has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-2cp2r", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-2cp2r has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-bnghx", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-bnghx has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-spjrm", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-spjrm has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-7gwqf", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-7gwqf has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-wzhd4", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-wzhd4 has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-z67pp", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-z67pp has been in a non-ready state for longer than 5 minutes. Firing 26: State: Resolved Started: 2026-10-18T22:30:02Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-8rjbk", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-8rjbk has been in a non-ready state for longer than 5 minutes. Firing 27: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-89fp5", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-89fp5 has been in a non-ready state for longer than 5 minutes. Firing 28: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-dfhsg", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-dfhsg has been in a non-ready state for longer than 5 minutes. Firing 29: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-ndv7w", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-ndv7w has been in a non-ready state for longer than 5 minutes. Firing 30: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-x65xf", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-x65xf has been in a non-ready state for longer than 5 minutes. Firing 31: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-9nhcz", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-9nhcz has been in a non-ready state for longer than 5 minutes. Firing 32: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-57rg5", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-57rg5 has been in a non-ready state for longer than 5 minutes. Firing 33: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-vqfkk", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-vqfkk has been in a non-ready state for longer than 5 minutes. Firing 34: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-mvbzv", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-mvbzv has been in a non-ready state for longer than 5 minutes. Firing 35: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-xqzzm", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-xqzzm has been in a non-ready state for longer than 5 minutes. Firing 36: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-pzz7r", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-pzz7r has been in a non-ready state for longer than 5 minutes. Firing 37: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:04Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-4ht2v", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-4ht2v has been in a non-ready state for longer than 5 minutes. Firing 38: State: Resolved Started: 2026-10-18T22:31:04Z Ended: 2026-10-18T22:55:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7176064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b", pod="network-node-identity-6585b9796c-m2n57", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2thfcg8puuokl14unu497m3b3fo4ib0u-q2l3y4m0x2b1n0b/network-node-identity-6585b9796c-m2n57 has been in a non-ready state for longer than 5 minutes. alert fired 7 time(s) (1 unknown, 6 known) Firing 1: State: Resolved Started: 2026-10-18T02:22:20Z Ended: 2026-10-18T02:28:23Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5039232-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgtj1aja7eu3kumpui0v9v0hrn0ctrn-n2t2k5k1d2q7b4m", pod="openshift-apiserver-74747bbdf5-l6sx2", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2tgtj1aja7eu3kumpui0v9v0hrn0ctrn-n2t2k5k1d2q7b4m/openshift-apiserver-74747bbdf5-l6sx2 has been in a non-ready state for longer than 5 minutes. alert fired 32 time(s) (25 unknown, 7 known) Firing 1: State: Resolved Started: 2026-10-17T09:53:56Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgf9u7qesjgmttof4tbj175kd9c3v6a-j2l3z7n5l1o3m5w", pod="network-node-identity-5479889bd6-wm9n5", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgf9u7qesjgmttof4tbj175kd9c3v6a-j2l3z7n5l1o3m5w/network-node-identity-5479889bd6-wm9n5 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-17T09:53:56Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgf9u7qesjgmttof4tbj175kd9c3v6a-j2l3z7n5l1o3m5w", pod="ovnkube-control-plane-6f598f876b-4lwmh", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgf9u7qesjgmttof4tbj175kd9c3v6a-j2l3z7n5l1o3m5w/ovnkube-control-plane-6f598f876b-4lwmh has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-17T09:53:57Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa0b8h25qc9a5i20tff4l0fg95ohb-x1q7f4j4x4u2t1g", pod="network-node-identity-5d7f9f545c-9fj88", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa0b8h25qc9a5i20tff4l0fg95ohb-x1q7f4j4x4u2t1g/network-node-identity-5d7f9f545c-9fj88 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-17T09:56:00Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa3i9d0dtsn3mqgm2m0jvbj0gost6-c8m7l6f8p4l9x4f", pod="ovnkube-control-plane-6776f795c7-pnvf4", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa3i9d0dtsn3mqgm2m0jvbj0gost6-c8m7l6f8p4l9x4f/ovnkube-control-plane-6776f795c7-pnvf4 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-17T09:56:00Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa3i9d0dtsn3mqgm2m0jvbj0gost6-c8m7l6f8p4l9x4f", pod="network-node-identity-6484b49989-tpf9j", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa3i9d0dtsn3mqgm2m0jvbj0gost6-c8m7l6f8p4l9x4f/network-node-identity-6484b49989-tpf9j has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-17T09:56:01Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacm1jgb7lcrs9ivlei7ps8sq33bk-w6k4h3j3b8i6v3d", pod="network-node-identity-77ddc9bb4c-pbwr7", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacm1jgb7lcrs9ivlei7ps8sq33bk-w6k4h3j3b8i6v3d/network-node-identity-77ddc9bb4c-pbwr7 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-17T09:56:01Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa5d10e98e3mrvr8mdtggsfec8ur7-z9i7c7u7w1a3k9w", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa5d10e98e3mrvr8mdtggsfec8ur7-z9i7c7u7w1a3k9w/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-17T09:56:01Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacm1jgb7lcrs9ivlei7ps8sq33bk-w6k4h3j3b8i6v3d", pod="ovnkube-control-plane-6bd64ff8dc-l5xp2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacm1jgb7lcrs9ivlei7ps8sq33bk-w6k4h3j3b8i6v3d/ovnkube-control-plane-6bd64ff8dc-l5xp2 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-17T09:58:01Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacmm0dlphe6vtc50mkmtd3jb2p8q-f2o6b4o3v3g6y5w", pod="kube-apiserver-9875495b7-tqsz4", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacmm0dlphe6vtc50mkmtd3jb2p8q-f2o6b4o3v3g6y5w/kube-apiserver-9875495b7-tqsz4 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-17T09:58:01Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacmm0dlphe6vtc50mkmtd3jb2p8q-f2o6b4o3v3g6y5w", pod="kube-apiserver-9875495b7-222sx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacmm0dlphe6vtc50mkmtd3jb2p8q-f2o6b4o3v3g6y5w/kube-apiserver-9875495b7-222sx has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-17T09:58:01Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfadvi8jv413ort809d0n4kq5kno22-s2q4c0i7f2c0r2x", pod="network-node-identity-5d8fccd895-ptmvr", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfadvi8jv413ort809d0n4kq5kno22-s2q4c0i7f2c0r2x/network-node-identity-5d8fccd895-ptmvr has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-17T09:58:01Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacmauure3rehr0ll3kf67lkonqn4-l7q9g3b3j1r5c8t", pod="network-node-identity-86d7bc77f4-zzxmb", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacmauure3rehr0ll3kf67lkonqn4-l7q9g3b3j1r5c8t/network-node-identity-86d7bc77f4-zzxmb has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-17T09:58:01Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacmm0dlphe6vtc50mkmtd3jb2p8q-f2o6b4o3v3g6y5w", pod="kube-apiserver-9875495b7-scjqk", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacmm0dlphe6vtc50mkmtd3jb2p8q-f2o6b4o3v3g6y5w/kube-apiserver-9875495b7-scjqk has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacoq552p37e3lj875ff1g01eve2p-j5s8a2u8s3o2v6k", pod="kube-apiserver-5844c8889f-cbz2x", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacoq552p37e3lj875ff1g01eve2p-j5s8a2u8s3o2v6k/kube-apiserver-5844c8889f-cbz2x has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacoq552p37e3lj875ff1g01eve2p-j5s8a2u8s3o2v6k", pod="kube-apiserver-5844c8889f-d6fvz", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacoq552p37e3lj875ff1g01eve2p-j5s8a2u8s3o2v6k/kube-apiserver-5844c8889f-d6fvz has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:05:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa3iukddj4rbptbdr63ko16fjd5a0-private-kas", pod="kube-apiserver-8544f679b7-zl64m", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa3iukddj4rbptbdr63ko16fjd5a0-private-kas/kube-apiserver-8544f679b7-zl64m has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa70s9eurd03qajc81h11klkgrmcp-q6y7b0q2i4d5g1m", pod="network-node-identity-6cd587db9f-t4z8h", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa70s9eurd03qajc81h11klkgrmcp-q6y7b0q2i4d5g1m/network-node-identity-6cd587db9f-t4z8h has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa3iukddj4rbptbdr63ko16fjd5a0-private-kas", pod="kube-apiserver-8544f679b7-5fcvk", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa3iukddj4rbptbdr63ko16fjd5a0-private-kas/kube-apiserver-8544f679b7-5fcvk has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfadpgoo9rvetrl85b24o6ppv4j0fo-o3i3s1q6c0r8x5c", pod="network-node-identity-5d97d9b5cb-4k88v", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfadpgoo9rvetrl85b24o6ppv4j0fo-o3i3s1q6c0r8x5c/network-node-identity-5d97d9b5cb-4k88v has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacn5q7o0r4p4o2djer1ifh59adg8-ea-listq25ppj", pod="network-node-identity-74f67fb4bf-ngwc7", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacn5q7o0r4p4o2djer1ifh59adg8-ea-listq25ppj/network-node-identity-74f67fb4bf-ngwc7 has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfacoq552p37e3lj875ff1g01eve2p-j5s8a2u8s3o2v6k", pod="kube-apiserver-5844c8889f-b42qs", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfacoq552p37e3lj875ff1g01eve2p-j5s8a2u8s3o2v6k/kube-apiserver-5844c8889f-b42qs has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfapf0dhnoj0q238ct23l9vk8nc5at-k7b2q6t5f3g5p6n", pod="network-node-identity-fb54b4d4d-v8wgh", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfapf0dhnoj0q238ct23l9vk8nc5at-k7b2q6t5f3g5p6n/network-node-identity-fb54b4d4d-v8wgh has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-10-17T09:58:58Z Ended: 2026-10-17T10:04:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfadpgoo9rvetrl85b24o6ppv4j0fo-o3i3s1q6c0r8x5c", pod="ovnkube-control-plane-6895bcbdc6-c9kvm", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfadpgoo9rvetrl85b24o6ppv4j0fo-o3i3s1q6c0r8x5c/ovnkube-control-plane-6895bcbdc6-c9kvm has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-10-17T09:58:59Z Ended: 2026-10-17T10:05:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa3iukddj4rbptbdr63ko16fjd5a0-private-kas", pod="kube-apiserver-8544f679b7-q8rkz", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa3iukddj4rbptbdr63ko16fjd5a0-private-kas/kube-apiserver-8544f679b7-q8rkz has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-10-17T09:59:55Z Ended: 2026-10-17T10:05:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0319872-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tgfa1d4f46ij8t9ghuvhfoa11l51bep-basic-cluster", pod="network-node-identity-5774b9669-l5ggc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tgfa1d4f46ij8t9ghuvhfoa11l51bep-basic-cluster/network-node-identity-5774b9669-l5ggc has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (37) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] LeaderElectionLeaseStale firedfull failure pattern: alert [svc] LeaderElectionLeaseStale fired Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | alert | 35 | 7.13%35 of 491 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 11: 1 · Oct 12: 5 · Oct 13: 13 · Oct 14: 8 · Oct 15: 9 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T21:26:40Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j4315776-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.179:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4315776/providers/microsoft.monitor/accounts/services-j4315776", namespace="klusterlet-2tfeeaom86b60erngar03ad51ehpgqtm", pod="arohcp-monitor-kube-state-metrics-7988dc648c-h4l8s", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tfeeaom86b60erngar03ad51ehpgqtm on cluster ci01-j4315776-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T18:05:49Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j7905664-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7905664/providers/microsoft.monitor/accounts/services-j7905664", namespace="klusterlet-2tfbjf51so3bj30gni3l73dohrjlrc07", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mz4bk", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tfbjf51so3bj30gni3l73dohrjlrc07 on cluster ci01-j7905664-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T18:00:18Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j3950080-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.234:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3950080/providers/microsoft.monitor/accounts/services-j3950080", namespace="klusterlet-2tfbf4h0d9nqbkbie85eg76q9fu1hc63", pod="arohcp-monitor-kube-state-metrics-7988dc648c-zrvv8", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tfbf4h0d9nqbkbie85eg76q9fu1hc63 on cluster ci01-j3950080-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. Contributing tests (1)
Affected runs (35) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [hcp] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 33 | 6.72%33 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 7 · Oct 13: 6 · Oct 14: 11 · Oct 15: 9 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T21:16:42Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2571392-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2571392/providers/microsoft.monitor/accounts/hcps-j2571392", namespace="ocm-arohcpci01-2tfehdfsiuk7rspp81ggshks0ldi46jk-d0c0o9z7l8b4h1m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mdkdv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfehdfsiuk7rspp81ggshks0ldi46jk-d0c0o9z7l8b4h1m/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-15T21:16:42Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2571392-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.203:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2571392/providers/microsoft.monitor/accounts/hcps-j2571392", namespace="ocm-arohcpci01-2tfehdfsiuk7rspp81ggshks0ldi46jk-d0c0o9z7l8b4h1m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mdkdv", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfehdfsiuk7rspp81ggshks0ldi46jk-d0c0o9z7l8b4h1m/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T19:40:29Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9990016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9990016/providers/microsoft.monitor/accounts/hcps-j9990016", namespace="ocm-arohcpci01-2tfcsk6itjfjpcjhjvmmra3bj97fh52v-d7p2j6h4b2a7d8x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-p222m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfcsk6itjfjpcjhjvmmra3bj97fh52v-d7p2j6h4b2a7d8x/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-15T19:41:30Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9990016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.118:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9990016/providers/microsoft.monitor/accounts/hcps-j9990016", namespace="ocm-arohcpci01-2tfcsk6itjfjpcjhjvmmra3bj97fh52v-d7p2j6h4b2a7d8x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-p222m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfcsk6itjfjpcjhjvmmra3bj97fh52v-d7p2j6h4b2a7d8x/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T18:44:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3520000-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.223:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3520000/providers/microsoft.monitor/accounts/hcps-j3520000", namespace="ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5k8tw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-15T18:44:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3520000-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.223:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3520000/providers/microsoft.monitor/accounts/hcps-j3520000", namespace="ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5k8tw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-15T18:44:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3520000-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.223:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3520000/providers/microsoft.monitor/accounts/hcps-j3520000", namespace="ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5k8tw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-15T18:44:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3520000-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.223:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3520000/providers/microsoft.monitor/accounts/hcps-j3520000", namespace="ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5k8tw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tfbvc1dj03nufhiu0qumf4o5mmkbqp1-w6h1y1s8u1u3w6q/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (33) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"full failure pattern: cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | e2e | 23 | 4.68%23 of 491 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 11: 0 · Oct 12: 5 · Oct 13: 6 · Oct 14: 7 · Oct 15: 4 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-9jbnpswbqhnc/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-9jbnpswbqhnc/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-kzwmzc2svmh2/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-kzwmzc2svmh2/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-fs95n68z9x97/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-fs95n68z9x97/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: FailedContributing tests (1)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | e2e | 21 | 4.28%21 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 11: 0 · Oct 12: 3 · Oct 13: 7 · Oct 14: 5 · Oct 15: 4 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3a1e5c39e140>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-lsp7k9q4q2pz" to finish creating: GET https://rp.j1762432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eeecd748-7cd9-428c-bdad-6172bba0e52f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eeecd748-7cd9-428c-bdad-6172bba0e52f",
"name": "eeecd748-7cd9-428c-bdad-6172bba0e52f",
"status": "Failed",
"startTime": "2026-10-16T17:39:40.172421101Z",
"endTime": "2026-10-16T17:58:48.49629868Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pb4xk-8sfrz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lsp7k9q4q2pz--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pb4xk-8sfrz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pb4xk-8sfrz: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-lsp7k9q4q2pz--managed/ingress-cert-le-np-1-pb4xk-8sfrz-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lsp7k9q4q2pz--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-pb4xk-8sfrz-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3871b527-fb1e-4123-b38b-3cb2445a9fc8' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1762432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eeecd748-7cd9-428c-bdad-6172bba0e52f
correlation ID: 3ee22f63-ca3a-4588-b9c5-ce97876471df
client request ID: a644f8be-e878-4c8d-99e9-321877a22983
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:163]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3a1e5c39e140>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-lsp7k9q4q2pz" to finish creating: GET https://rp.j1762432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eeecd748-7cd9-428c-bdad-6172bba0e52f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eeecd748-7cd9-428c-bdad-6172bba0e52f",
"name": "eeecd748-7cd9-428c-bdad-6172bba0e52f",
"status": "Failed",
"startTime": "2026-10-16T17:39:40.172421101Z",
"endTime": "2026-10-16T17:58:48.49629868Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-pb4xk-8sfrz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lsp7k9q4q2pz--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-pb4xk-8sfrz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-pb4xk-8sfrz: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-lsp7k9q4q2pz--managed/ingress-cert-le-np-1-pb4xk-8sfrz-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lsp7k9q4q2pz--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-pb4xk-8sfrz-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3871b527-fb1e-4123-b38b-3cb2445a9fc8' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1762432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/eeecd748-7cd9-428c-bdad-6172bba0e52f
correlation ID: 3ee22f63-ca3a-4588-b9c5-ce97876471df
client request ID: a644f8be-e878-4c8d-99e9-321877a22983fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3d49e0c9b600>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cqn2ccl8wpmh" to finish creating: GET https://rp.j3090304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8",
"name": "57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8",
"status": "Failed",
"startTime": "2026-10-16T11:01:48.232534133Z",
"endTime": "2026-10-16T11:20:52.453260342Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0j2o7l0q7m7m7f-np-delete-main-4ql8g-j89nf: NodeDoesNotExist: Node does not exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0j2o7l0q7m7m7f-np-delete-main-4ql8g-j89nf: NotReady: * Deleting: Machine deletion in progress, stage: WaitingForInfrastructureDeletion\n* HealthCheckSucceeded: Health check failed: Node failed to report startup in 10m0s\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3090304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8
correlation ID: f387d1ed-05da-402f-8ada-b632f0d02aae
client request ID: 36e27b8d-f92f-4579-9e74-48bd79aecb10
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3d49e0c9b600>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-cqn2ccl8wpmh" to finish creating: GET https://rp.j3090304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8",
"name": "57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8",
"status": "Failed",
"startTime": "2026-10-16T11:01:48.232534133Z",
"endTime": "2026-10-16T11:20:52.453260342Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j0j2o7l0q7m7m7f-np-delete-main-4ql8g-j89nf: NodeDoesNotExist: Node does not exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j0j2o7l0q7m7m7f-np-delete-main-4ql8g-j89nf: NotReady: * Deleting: Machine deletion in progress, stage: WaitingForInfrastructureDeletion\n* HealthCheckSucceeded: Health check failed: Node failed to report startup in 10m0s\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3090304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57ef0bb1-ee3a-46b4-87ee-9e92b66bb5c8
correlation ID: f387d1ed-05da-402f-8ada-b632f0d02aae
client request ID: 36e27b8d-f92f-4579-9e74-48bd79aecb10fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x31dfe7cead80>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-w4sdcc2f9b98" to finish creating: GET https://rp.j9466368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1a9a190b-4bfc-40cb-9dd3-61f444b3d45a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1a9a190b-4bfc-40cb-9dd3-61f444b3d45a",
"name": "1a9a190b-4bfc-40cb-9dd3-61f444b3d45a",
"status": "Failed",
"startTime": "2026-10-15T11:58:54.751404091Z",
"endTime": "2026-10-15T12:28:03.314684153Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-w4sdcc2f9b98--managed/providers/Microsoft.Compute/virtualMachines/v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-w4sdcc2f9b98--managed/providers/Microsoft.Compute/virtualMachines/v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9466368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1a9a190b-4bfc-40cb-9dd3-61f444b3d45a
correlation ID: a5182798-ac67-4625-a997-1317fca7e220
client request ID: d5f9fdbc-cb7e-4972-bee6-661a76843384
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x31dfe7cead80>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-w4sdcc2f9b98" to finish creating: GET https://rp.j9466368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1a9a190b-4bfc-40cb-9dd3-61f444b3d45a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1a9a190b-4bfc-40cb-9dd3-61f444b3d45a",
"name": "1a9a190b-4bfc-40cb-9dd3-61f444b3d45a",
"status": "Failed",
"startTime": "2026-10-15T11:58:54.751404091Z",
"endTime": "2026-10-15T12:28:03.314684153Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-w4sdcc2f9b98--managed/providers/Microsoft.Compute/virtualMachines/v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-w4sdcc2f9b98--managed/providers/Microsoft.Compute/virtualMachines/v0x6h3c4b5g9z6x-np-delete-main-cv9ln-97hs2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9466368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1a9a190b-4bfc-40cb-9dd3-61f444b3d45a
correlation ID: a5182798-ac67-4625-a997-1317fca7e220
client request ID: d5f9fdbc-cb7e-4972-bee6-661a76843384Contributing tests (4)
Affected runs (21) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | e2e | 10 | 2.04%10 of 491 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 3 · Oct 13: 6 · Oct 14: 3 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-6xdv46kj2t6c/cluster-candidate-4-23-ds6g6p should provision
Unexpected error:
<*fmt.wrapError | 0x100c616f6120>:
failed to create HCP cluster cluster-candidate-4-23-ds6g6p: failed waiting for cluster="cluster-candidate-4-23-ds6g6p" in resourcegroup="rg-candidate-4-23-6xdv46kj2t6c" to finish creating: GET https://rp.j0840320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f51531d5-d416-4611-a4d7-f0f2d7392927
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f51531d5-d416-4611-a4d7-f0f2d7392927",
"name": "f51531d5-d416-4611-a4d7-f0f2d7392927",
"status": "Failed",
"startTime": "2026-10-15T17:08:33.97123397Z",
"endTime": "2026-10-15T17:37:33.800671635Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0840320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f51531d5-d416-4611-a4d7-f0f2d7392927
correlation ID: b723ee7e-f41a-48af-8379-6cf9b8adcb05
client request ID: a255119f-4ee0-4d8e-8cd6-e4350ddd6c3d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-6xdv46kj2t6c/cluster-candidate-4-23-ds6g6p should provision
Unexpected error:
<*fmt.wrapError | 0x100c616f6120>:
failed to create HCP cluster cluster-candidate-4-23-ds6g6p: failed waiting for cluster="cluster-candidate-4-23-ds6g6p" in resourcegroup="rg-candidate-4-23-6xdv46kj2t6c" to finish creating: GET https://rp.j0840320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f51531d5-d416-4611-a4d7-f0f2d7392927
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f51531d5-d416-4611-a4d7-f0f2d7392927",
"name": "f51531d5-d416-4611-a4d7-f0f2d7392927",
"status": "Failed",
"startTime": "2026-10-15T17:08:33.97123397Z",
"endTime": "2026-10-15T17:37:33.800671635Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0840320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f51531d5-d416-4611-a4d7-f0f2d7392927
correlation ID: b723ee7e-f41a-48af-8379-6cf9b8adcb05
client request ID: a255119f-4ee0-4d8e-8cd6-e4350ddd6c3d
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x24524eb9ea50>:
GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
"name": "812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
"status": "Failed",
"startTime": "2026-10-14T23:04:33.551295422Z",
"endTime": "2026-10-14T23:33:41.530060174Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
correlation ID: d14d5694-03de-4ac8-b587-4d0e35762d84
client request ID: 5cfd8941-3a60-4ebe-8bc6-a659901f1339
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x24524eb9ea50>:
GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
"name": "812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
"status": "Failed",
"startTime": "2026-10-14T23:04:33.551295422Z",
"endTime": "2026-10-14T23:33:41.530060174Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
correlation ID: d14d5694-03de-4ac8-b587-4d0e35762d84
client request ID: 5cfd8941-3a60-4ebe-8bc6-a659901f1339
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x225a34843540>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5rp2rtlm9fj9" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619",
"name": "680a3372-b50b-4c21-a07b-0256c4145619",
"status": "Failed",
"startTime": "2026-10-14T09:49:54.805487688Z",
"endTime": "2026-10-14T10:18:57.194996434Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
correlation ID: 5895723a-3ff0-431a-a927-5b6e194fc741
client request ID: 350b7a37-c335-451c-b2e9-99a3b665afa1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x225a34843540>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5rp2rtlm9fj9" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619",
"name": "680a3372-b50b-4c21-a07b-0256c4145619",
"status": "Failed",
"startTime": "2026-10-14T09:49:54.805487688Z",
"endTime": "2026-10-14T10:18:57.194996434Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
correlation ID: 5895723a-3ff0-431a-a927-5b6e194fc741
client request ID: 350b7a37-c335-451c-b2e9-99a3b665afa1
occurredContributing tests (12)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodCrashLooping firedfull failure pattern: alert [hcp] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 10 | 2.04%10 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 4 · Oct 13: 2 · Oct 14: 2 · Oct 15: 2 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-15T17:33:59Z Ended: 2026-10-15T18:00:59Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0840320-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.184:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0840320/providers/microsoft.monitor/accounts/hcps-j0840320", namespace="ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="69abfa76-e157-4c11-8f36-2d5d03c40035" Description: Pod ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-15T17:33:59Z Ended: 2026-10-15T17:59:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0840320-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.184:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0840320/providers/microsoft.monitor/accounts/hcps-j0840320", namespace="ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="69abfa76-e157-4c11-8f36-2d5d03c40035" Description: Pod ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T04:10:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7932288-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.98:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7932288/providers/microsoft.monitor/accounts/hcps-j7932288", namespace="ocm-arohcpci01-2tevnsff1cm4oc58rjaeu9o2afr8b41s-o1p7c9y0n1v9w4w", pod="cloud-network-config-controller-5dbf455969-n498c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="724a7794-4308-471d-8b59-9caf89cd21b6" Description: Pod ocm-arohcpci01-2tevnsff1cm4oc58rjaeu9o2afr8b41s-o1p7c9y0n1v9w4w/cloud-network-config-controller-5dbf455969-n498c (controller) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-15T04:10:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7932288-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.98:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7932288/providers/microsoft.monitor/accounts/hcps-j7932288", namespace="ocm-arohcpci01-2tevnsff1cm4oc58rjaeu9o2afr8b41s-o1p7c9y0n1v9w4w", pod="cloud-network-config-controller-5dbf455969-n498c", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="724a7794-4308-471d-8b59-9caf89cd21b6" Description: Pod ocm-arohcpci01-2tevnsff1cm4oc58rjaeu9o2afr8b41s-o1p7c9y0n1v9w4w/cloud-network-config-controller-5dbf455969-n498c (controller) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-14T23:26:27Z Ended: 2026-10-14T23:56:29Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e7a6870f-14b4-45e0-9c19-fce7f03813dc" Description: Pod ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-14T23:26:27Z Ended: 2026-10-14T23:56:29Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e7a6870f-14b4-45e0-9c19-fce7f03813dc" Description: Pod ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 9 | 1.83%9 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 3 · Oct 13: 3 · Oct 14: 2 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-15T17:31:05Z Ended: 2026-10-15T17:56:03Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0840320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.184:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0840320/providers/microsoft.monitor/accounts/hcps-j0840320", namespace="ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5z7k7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-15T17:31:05Z Ended: 2026-10-15T18:01:04Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0840320-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.184:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0840320/providers/microsoft.monitor/accounts/hcps-j0840320", namespace="ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5z7k7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tfbl4gqpolf3q5urat1gqkjg77m4lg4-a9g3b1n2x3r1g8j/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-14T23:34:33Z Ended: 2026-10-14T23:51:31Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r2hbm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-14T23:35:30Z Ended: 2026-10-14T23:52:28Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r2hbm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-14T10:09:45Z Ended: 2026-10-14T10:36:51Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1691264-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.235:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1691264/providers/microsoft.monitor/accounts/hcps-j1691264", namespace="ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9qldk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-14T10:09:45Z Ended: 2026-10-14T10:36:51Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1691264-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.235:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1691264/providers/microsoft.monitor/accounts/hcps-j1691264", namespace="ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9qldk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] MaestroAgentMetricsProxyDown firedfull failure pattern: alert [svc] MaestroAgentMetricsProxyDown fired Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | alert | 8 | 1.63%8 of 491 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 2 · Oct 15: 3 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-16T15:54:14Z Ended: 2026-10-16T16:02:12Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j7660416-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. Firing 2: State: Resolved Started: 2026-10-16T15:55:11Z Ended: 2026-10-16T16:02:12Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j7660416-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-16T14:26:06Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j8714496-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T17:43:36Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4658048-mgmt-2", component="monitoring-infrastructure", region="centralus", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 8 | 1.63%8 of 491 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 11: 0 · Oct 12: 2 · Oct 13: 0 · Oct 14: 2 · Oct 15: 4 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2d72cc8aa180>:
resource group "autoscaling-cluster-7hjmhzg78tkl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7hjmhzg78tkl" to finish deleting: GET https://rp.j3520000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c3202962-b736-4f51-840b-2c3b32fd9172
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c3202962-b736-4f51-840b-2c3b32fd9172",
"name": "c3202962-b736-4f51-840b-2c3b32fd9172",
"status": "Failed",
"startTime": "2026-10-15T18:00:50.353777Z",
"endTime": "2026-10-15T18:24:53.714360344Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tfbvc1dj03nufhiu0qumf4o5mmkbqp1 still exists (deletion dispatched at 2026-10-15T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3520000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c3202962-b736-4f51-840b-2c3b32fd9172
correlation ID: 2ab924cd-c6a5-4535-afa1-4285e0d7421a
client request ID: 05581b93-1450-43ab-a2a8-826d6a1faeda
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2d72cc8aa180>:
resource group "autoscaling-cluster-7hjmhzg78tkl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7hjmhzg78tkl" to finish deleting: GET https://rp.j3520000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c3202962-b736-4f51-840b-2c3b32fd9172
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c3202962-b736-4f51-840b-2c3b32fd9172",
"name": "c3202962-b736-4f51-840b-2c3b32fd9172",
"status": "Failed",
"startTime": "2026-10-15T18:00:50.353777Z",
"endTime": "2026-10-15T18:24:53.714360344Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tfbvc1dj03nufhiu0qumf4o5mmkbqp1 still exists (deletion dispatched at 2026-10-15T18:00:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3520000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c3202962-b736-4f51-840b-2c3b32fd9172
correlation ID: 2ab924cd-c6a5-4535-afa1-4285e0d7421a
client request ID: 05581b93-1450-43ab-a2a8-826d6a1faeda
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0xb39be402570>:
resource group "rg-zstream-upgrade-4-22-qkz2pwtj5n7j": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-22-9x7njw" in resourcegroup="rg-zstream-upgrade-4-22-qkz2pwtj5n7j" to finish deleting: GET https://rp.j0325120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a",
"name": "a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a",
"status": "Failed",
"startTime": "2026-10-15T17:05:09.736854856Z",
"endTime": "2026-10-15T17:29:18.684928477Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tfb63s18q5cesobi99a7b7bb7k3a72n still exists (deletion dispatched at 2026-10-15T17:05:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0325120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a
correlation ID: ce6f6715-7b02-4412-b5e3-4934b1a27812
client request ID: bf6cd98a-47f4-43d5-af65-2ee26c5ddfb6
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0xb39be402570>:
resource group "rg-zstream-upgrade-4-22-qkz2pwtj5n7j": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-22-9x7njw" in resourcegroup="rg-zstream-upgrade-4-22-qkz2pwtj5n7j" to finish deleting: GET https://rp.j0325120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a",
"name": "a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a",
"status": "Failed",
"startTime": "2026-10-15T17:05:09.736854856Z",
"endTime": "2026-10-15T17:29:18.684928477Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tfb63s18q5cesobi99a7b7bb7k3a72n still exists (deletion dispatched at 2026-10-15T17:05:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0325120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a5d1c8a5-60e3-4c19-acaf-41b8b35cb86a
correlation ID: ce6f6715-7b02-4412-b5e3-4934b1a27812
client request ID: bf6cd98a-47f4-43d5-af65-2ee26c5ddfb6
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0xa89f6d94528>:
resource group "rg-np-noedge-tw7cjj-xlndscvj7dwp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-tw7cjj" in resourcegroup="rg-np-noedge-tw7cjj-xlndscvj7dwp" to finish deleting: GET https://rp.j5616384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/19704973-55a6-47ab-b561-11bb7cb66fb8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/19704973-55a6-47ab-b561-11bb7cb66fb8",
"name": "19704973-55a6-47ab-b561-11bb7cb66fb8",
"status": "Failed",
"startTime": "2026-10-15T14:51:30.063016611Z",
"endTime": "2026-10-15T15:15:36.942553753Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tf9cp7cug191vvs2krpcsqb89jucjnj still exists (deletion dispatched at 2026-10-15T14:51:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5616384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/19704973-55a6-47ab-b561-11bb7cb66fb8
correlation ID: 6de0dbef-67fc-48b2-ad3a-a51b2af3faa1
client request ID: a410cfe8-06c4-4eb7-afde-a18a2ead7f72
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0xa89f6d94528>:
resource group "rg-np-noedge-tw7cjj-xlndscvj7dwp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-noedge-tw7cjj" in resourcegroup="rg-np-noedge-tw7cjj-xlndscvj7dwp" to finish deleting: GET https://rp.j5616384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/19704973-55a6-47ab-b561-11bb7cb66fb8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/19704973-55a6-47ab-b561-11bb7cb66fb8",
"name": "19704973-55a6-47ab-b561-11bb7cb66fb8",
"status": "Failed",
"startTime": "2026-10-15T14:51:30.063016611Z",
"endTime": "2026-10-15T15:15:36.942553753Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tf9cp7cug191vvs2krpcsqb89jucjnj still exists (deletion dispatched at 2026-10-15T14:51:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5616384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/19704973-55a6-47ab-b561-11bb7cb66fb8
correlation ID: 6de0dbef-67fc-48b2-ad3a-a51b2af3faa1
client request ID: a410cfe8-06c4-4eb7-afde-a18a2ead7f72
occurredContributing tests (8)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | alert | 7 | 1.43%7 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 11: 0 · Oct 12: 2 · Oct 13: 1 · Oct 14: 1 · Oct 15: 2 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 18 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-16T16:03:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-c4d6b786-548pt", region="westus3", severity="warning" Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4d6b786-548pt has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-5lt2p", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-5lt2p has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-nw2d2", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-nw2d2 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-rsnz7", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-rsnz7 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-q6lgt", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-q6lgt has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-27216613-vmss000000-jg5jx", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-27216613-vmss000000-jg5jx has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-58594b5dd7-c85j5", region="westus3", severity="warning" Description: Pod kube-system/hubble-relay-58594b5dd7-c85j5 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-9j44l", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-9j44l has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-27216613-vmss000002-wgh5j", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-27216613-vmss000002-wgh5j has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-27216613-vmss000001-zvc5c", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-27216613-vmss000001-zvc5c has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-16T16:05:01Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-bjvb8", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-bjvb8 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6cb67d7f44-f4tbd", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-6cb67d7f44-f4tbd has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-77fbb4f5f5-2fjpq", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-77fbb4f5f5-2fjpq has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6cb67d7f44-89fpd", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-6cb67d7f44-89fpd has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-6cb67d7f44-g6qp6", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-6cb67d7f44-g6qp6 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-7dfb5b7d6b-dgw7q", region="westus3", severity="warning" Description: Pod maestro/maestro-7dfb5b7d6b-dgw7q has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-7dfb5b7d6b-qws2j", region="westus3", severity="warning" Description: Pod maestro/maestro-7dfb5b7d6b-qws2j has been in a non-ready state for longer than 5 minutes. Firing 18: State: Fired (not resolved) Started: 2026-10-16T16:13:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7660416-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-77fbb4f5f5-gdqfm", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-77fbb4f5f5-gdqfm has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T20:18:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5701632-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-c4d6b786-v5xlx", region="centralus", severity="warning" Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-c4d6b786-v5xlx has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T17:40:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4658048-mgmt-1", component="kubernetes-infrastructure", namespace="kube-system", pod="set-kubelet-parameters-for-scale-r7jxg", region="centralus", severity="warning" Description: Pod kube-system/set-kubelet-parameters-for-scale-r7jxg has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 7 | 1.43%7 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 5 · Oct 15: 1 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x8a369199c00>:
DNS for route host agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x8a36a1383e0>{
msg: "DNS for agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x8a369199c00>:
DNS for route host agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x8a36a1383e0>{
msg: "DNS for agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2qggj.apps.aro.v20260901.29of.j9678720.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-5bxn7w"
Unexpected error:
<*fmt.wrapError | 0x37da2d0209a0>:
DNS for route host agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x37da2d0168c0>{
msg: "DNS for agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-5bxn7w"
Unexpected error:
<*fmt.wrapError | 0x37da2d0209a0>:
DNS for route host agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x37da2d0168c0>{
msg: "DNS for agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-v5x65.apps.aro.j6r3d2d0n3v4o5x.zbh9.j2571392.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1fd36b60420>:
DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1fd36b60400>{
msg: "DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1fd36b60420>:
DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1fd36b60400>{
msg: "DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (4)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 6 | 1.22%6 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 2 · Oct 14: 1 · Oct 15: 2 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:365]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x26aec5b792f0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40106->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40106->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:365]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x26aec5b792f0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40106->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40106->172.30.0.10:53: read: connection refused",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:365]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2b9d750c1520>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58152->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58152->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:365]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2b9d750c1520>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58152->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58152->172.30.0.10:53: read: connection refused",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:365]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3f466db88d40>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:33148->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:33148->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:365]: Timed out after 300.006s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x3f466db88d40>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:33148->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:33148->172.30.0.10:53: read: connection refused",
}Contributing tests (2)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 5 | 1.02%5 of 491 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 2 · Oct 15: 1 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x248dbdd6a2d0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-p9pjc5tph49f" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-p9pjc5tph49f, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-p9pjc5tph49f\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-p9pjc5tph49f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x248dbde747e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-p9pjc5tph49f",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x248dbdd6a2d0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-p9pjc5tph49f" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-p9pjc5tph49f, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-p9pjc5tph49f\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-p9pjc5tph49f, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x248dbde747e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-p9pjc5tph49f",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x3e6f09e44c00>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-k7gbzwzskr9p" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-k7gbzwzskr9p, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-k7gbzwzskr9p\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-k7gbzwzskr9p, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3e6f0a152150>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-k7gbzwzskr9p",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x3e6f09e44c00>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-k7gbzwzskr9p" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-k7gbzwzskr9p, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-k7gbzwzskr9p\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-k7gbzwzskr9p, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3e6f0a152150>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-k7gbzwzskr9p",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x1399efb46de0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-2n7cffz9j9vb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-2n7cffz9j9vb\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1399ef809d30>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x1399efb46de0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-2n7cffz9j9vb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-2n7cffz9j9vb\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1399ef809d30>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| take 101, Post "<url>": context deadline exceededfull failure pattern: | take 101, Post "<url>": context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history | e2e | 57 | 11.61%57 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 2 · Oct 14: 26 · Oct 15: 18 · Oct 16: 10 · Oct 17: 1 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x35a206020900>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-17T22:37:32.6448555Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-17T22:37:32.6448555Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x35a2060208e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-17T22:37:32.6448555Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x35a2051a8cd0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x35a2060208c0>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-17T22:37:32.6448555Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x35a206b44300>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x35a206020900>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-17T22:37:32.6448555Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-17T22:37:32.6448555Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x35a2060208e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-17T22:37:32.6448555Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x35a2051a8cd0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x35a2060208c0>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-x6nbt9kxc5r4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-17T22:37:32.6448555Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x35a206b44300>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x225983c28ba0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-16T15:59:20.1954973Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T15:59:20.1954973Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x225983c28b60>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T15:59:20.1954973Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x225983a80280>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x225983c28b20>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T15:59:20.1954973Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x22598544d170>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x225983c28ba0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-16T15:59:20.1954973Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T15:59:20.1954973Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x225983c28b60>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T15:59:20.1954973Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x225983a80280>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x225983c28b20>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-69hzvgr8h68z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T15:59:20.1954973Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x22598544d170>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2a4a4982c560>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-16T14:33:12.4130712Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T14:33:12.4130712Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x2a4a4982c540>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T14:33:12.4130712Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2a4a477c0190>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2a4a4982c520>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T14:33:12.4130712Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2a4a49889830>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2a4a4982c560>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-16T14:33:12.4130712Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T14:33:12.4130712Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x2a4a4982c540>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T14:33:12.4130712Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2a4a477c0190>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2a4a4982c520>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-97wmvqc47hmc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-16T14:33:12.4130712Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2a4a49889830>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredContributing tests (1)
Affected runs (57) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history | e2e | 16 | 3.26%16 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 2 · Oct 13: 3 · Oct 14: 2 · Oct 15: 7 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 | PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5mfh5m" for cluster "cluster-candidate-4-23-5mfh5m"
Unexpected error:
<*fmt.wrapError | 0x83c94b64840>:
failed to create NodePool np-5mfh5m: failed waiting for nodepool="np-5mfh5m" for cluster "cluster-candidate-4-23-5mfh5m" in resourcegroup="rg-candidate-4-23-t8lk4npslqvb" to finish creating: GET https://rp.j3075072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4ba6beec-b4a8-4b19-9703-a8d4018563a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4ba6beec-b4a8-4b19-9703-a8d4018563a9",
"name": "4ba6beec-b4a8-4b19-9703-a8d4018563a9",
"status": "Failed",
"startTime": "2026-10-16T10:16:52.661682642Z",
"endTime": "2026-10-16T10:35:56.734294404Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t8lk4npslqvb-candidate-5mfh5m--managed/providers/Microsoft.Compute/virtualMachines/w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t8lk4npslqvb-candidate-5mfh5m--managed/providers/Microsoft.Compute/virtualMachines/w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3075072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4ba6beec-b4a8-4b19-9703-a8d4018563a9
correlation ID: 2de83bb6-2187-443d-9a87-45e45c14da85
client request ID: 229ecc64-817b-4213-bc11-c707f778c1d6
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5mfh5m" for cluster "cluster-candidate-4-23-5mfh5m"
Unexpected error:
<*fmt.wrapError | 0x83c94b64840>:
failed to create NodePool np-5mfh5m: failed waiting for nodepool="np-5mfh5m" for cluster "cluster-candidate-4-23-5mfh5m" in resourcegroup="rg-candidate-4-23-t8lk4npslqvb" to finish creating: GET https://rp.j3075072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4ba6beec-b4a8-4b19-9703-a8d4018563a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4ba6beec-b4a8-4b19-9703-a8d4018563a9",
"name": "4ba6beec-b4a8-4b19-9703-a8d4018563a9",
"status": "Failed",
"startTime": "2026-10-16T10:16:52.661682642Z",
"endTime": "2026-10-16T10:35:56.734294404Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t8lk4npslqvb-candidate-5mfh5m--managed/providers/Microsoft.Compute/virtualMachines/w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-t8lk4npslqvb-candidate-5mfh5m--managed/providers/Microsoft.Compute/virtualMachines/w2t3o1i6q4w1u2r-np-5mfh5m-477cf-hfl4n to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3075072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4ba6beec-b4a8-4b19-9703-a8d4018563a9
correlation ID: 2de83bb6-2187-443d-9a87-45e45c14da85
client request ID: 229ecc64-817b-4213-bc11-c707f778c1d6
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x3408412f86c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nvxdbn5qmgtb" to finish creating: GET https://rp.j6657920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a18fbb87-8bb7-4a74-8f21-e96af9bfcf16
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a18fbb87-8bb7-4a74-8f21-e96af9bfcf16",
"name": "a18fbb87-8bb7-4a74-8f21-e96af9bfcf16",
"status": "Failed",
"startTime": "2026-10-16T08:39:54.399138921Z",
"endTime": "2026-10-16T08:58:56.325845484Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-nvxdbn5qmgtb--managed/providers/Microsoft.Compute/virtualMachines/p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-nvxdbn5qmgtb--managed/providers/Microsoft.Compute/virtualMachines/p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6657920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a18fbb87-8bb7-4a74-8f21-e96af9bfcf16
correlation ID: 956ae344-8430-4fa1-b7b7-d3523f2ec507
client request ID: 149dc074-a954-4585-b9e7-50faa51cea7a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x3408412f86c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nvxdbn5qmgtb" to finish creating: GET https://rp.j6657920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a18fbb87-8bb7-4a74-8f21-e96af9bfcf16
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a18fbb87-8bb7-4a74-8f21-e96af9bfcf16",
"name": "a18fbb87-8bb7-4a74-8f21-e96af9bfcf16",
"status": "Failed",
"startTime": "2026-10-16T08:39:54.399138921Z",
"endTime": "2026-10-16T08:58:56.325845484Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-nvxdbn5qmgtb--managed/providers/Microsoft.Compute/virtualMachines/p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-nvxdbn5qmgtb--managed/providers/Microsoft.Compute/virtualMachines/p8r1u9p3j7r2d5c-np-1-gkb8v-jqcmv to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6657920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a18fbb87-8bb7-4a74-8f21-e96af9bfcf16
correlation ID: 956ae344-8430-4fa1-b7b7-d3523f2ec507
client request ID: 149dc074-a954-4585-b9e7-50faa51cea7a
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x39a645494420>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-tfxpkjkjrqvw" to finish creating: GET https://rp.j8228736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54644c94-7f95-4829-9f03-e9b939189ba9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54644c94-7f95-4829-9f03-e9b939189ba9",
"name": "54644c94-7f95-4829-9f03-e9b939189ba9",
"status": "Failed",
"startTime": "2026-10-15T15:18:30.534168337Z",
"endTime": "2026-10-15T15:47:34.762349229Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-tfxpkjkjrqvw--managed/providers/Microsoft.Compute/virtualMachines/q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-tfxpkjkjrqvw--managed/providers/Microsoft.Compute/virtualMachines/q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8228736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54644c94-7f95-4829-9f03-e9b939189ba9
correlation ID: af51592b-1bd9-4ddd-81a5-d7c702c77eb4
client request ID: e54f4800-8170-46d4-a0fb-e016b4fc8317
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x39a645494420>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-tfxpkjkjrqvw" to finish creating: GET https://rp.j8228736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54644c94-7f95-4829-9f03-e9b939189ba9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54644c94-7f95-4829-9f03-e9b939189ba9",
"name": "54644c94-7f95-4829-9f03-e9b939189ba9",
"status": "Failed",
"startTime": "2026-10-15T15:18:30.534168337Z",
"endTime": "2026-10-15T15:47:34.762349229Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-tfxpkjkjrqvw--managed/providers/Microsoft.Compute/virtualMachines/q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-tfxpkjkjrqvw--managed/providers/Microsoft.Compute/virtualMachines/q2n5p1l1g7q3y7a-np-1-6mdqn-p62bb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8228736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/54644c94-7f95-4829-9f03-e9b939189ba9
correlation ID: af51592b-1bd9-4ddd-81a5-d7c702c77eb4
client request ID: e54f4800-8170-46d4-a0fb-e016b4fc8317
occurredContributing tests (6)
Affected runs (16) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s) | e2e | 11 | 2.24%11 of 491 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s) | Show trend detailsOct 11: 1 · Oct 12: 1 · Oct 13: 1 · Oct 14: 4 · Oct 15: 5 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x340dc62fa018>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-dsg52jhcqb45" to finish creating: GET https://rp.j7338752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baecf025-dfba-4e06-b1e1-28ab3b4cbfe6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baecf025-dfba-4e06-b1e1-28ab3b4cbfe6",
"name": "baecf025-dfba-4e06-b1e1-28ab3b4cbfe6",
"status": "Failed",
"startTime": "2026-10-16T14:26:03.913560661Z",
"endTime": "2026-10-16T14:45:08.626305681Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7338752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baecf025-dfba-4e06-b1e1-28ab3b4cbfe6
correlation ID: 85e240f2-d0e4-4c29-bd30-d4d4327881c2
client request ID: d5c7b389-b2c6-4aba-9dfa-9055479c941f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x340dc62fa018>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-dsg52jhcqb45" to finish creating: GET https://rp.j7338752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baecf025-dfba-4e06-b1e1-28ab3b4cbfe6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baecf025-dfba-4e06-b1e1-28ab3b4cbfe6",
"name": "baecf025-dfba-4e06-b1e1-28ab3b4cbfe6",
"status": "Failed",
"startTime": "2026-10-16T14:26:03.913560661Z",
"endTime": "2026-10-16T14:45:08.626305681Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7338752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/baecf025-dfba-4e06-b1e1-28ab3b4cbfe6
correlation ID: 85e240f2-d0e4-4c29-bd30-d4d4327881c2
client request ID: d5c7b389-b2c6-4aba-9dfa-9055479c941f
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0x12c08c2c7ca0>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ftltmptlj2wv" to finish creating: GET https://rp.j5160960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a570a939-3665-4ef5-997f-a303cdd226fe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a570a939-3665-4ef5-997f-a303cdd226fe",
"name": "a570a939-3665-4ef5-997f-a303cdd226fe",
"status": "Failed",
"startTime": "2026-10-15T21:24:37.707389453Z",
"endTime": "2026-10-15T21:53:37.621733074Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5160960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a570a939-3665-4ef5-997f-a303cdd226fe
correlation ID: f3ada34b-c475-444b-90fd-a47f28fcd020
client request ID: 8db3b288-30a6-4fd1-8aaf-660ebcafd494
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0x12c08c2c7ca0>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-ftltmptlj2wv" to finish creating: GET https://rp.j5160960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a570a939-3665-4ef5-997f-a303cdd226fe
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a570a939-3665-4ef5-997f-a303cdd226fe",
"name": "a570a939-3665-4ef5-997f-a303cdd226fe",
"status": "Failed",
"startTime": "2026-10-15T21:24:37.707389453Z",
"endTime": "2026-10-15T21:53:37.621733074Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5160960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a570a939-3665-4ef5-997f-a303cdd226fe
correlation ID: f3ada34b-c475-444b-90fd-a47f28fcd020
client request ID: 8db3b288-30a6-4fd1-8aaf-660ebcafd494
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x16cbbb3841e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-qc5d6jswb5qj" to finish creating: GET https://rp.j9447936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc6a40ed-7014-400d-bad8-400eddcb5f30
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc6a40ed-7014-400d-bad8-400eddcb5f30",
"name": "dc6a40ed-7014-400d-bad8-400eddcb5f30",
"status": "Failed",
"startTime": "2026-10-15T18:11:06.765137924Z",
"endTime": "2026-10-15T18:40:10.370410387Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9447936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc6a40ed-7014-400d-bad8-400eddcb5f30
correlation ID: 36d679d4-7225-44c2-8efc-12d1c6c12f83
client request ID: 0fff6042-c3bb-454b-84b5-80c6919f7c90
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x16cbbb3841e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-qc5d6jswb5qj" to finish creating: GET https://rp.j9447936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc6a40ed-7014-400d-bad8-400eddcb5f30
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc6a40ed-7014-400d-bad8-400eddcb5f30",
"name": "dc6a40ed-7014-400d-bad8-400eddcb5f30",
"status": "Failed",
"startTime": "2026-10-15T18:11:06.765137924Z",
"endTime": "2026-10-15T18:40:10.370410387Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9447936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc6a40ed-7014-400d-bad8-400eddcb5f30
correlation ID: 36d679d4-7225-44c2-8efc-12d1c6c12f83
client request ID: 0fff6042-c3bb-454b-84b5-80c6919f7c90
occurredContributing tests (10)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | provision | 10 | 2.04%10 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 9 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-16T10:37:16.131Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-16T10:37:17.576Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-16T10:37:18.879Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=182dddaef1eb2eeeaeb1c74f26cc60d61fef94e53fb090c30b83b72986af948d portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6554112-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F182dddaef1eb2eeeaeb1c74f26cc60d61fef94e53fb090c30b83b72986af948d
time=2026-10-16T10:39:19.372Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6554112-svc/providers/Microsoft.Resources/deployments/182dddaef1eb2eeeaeb1c74f26cc60d61fef94e53fb090c30b83b72986af948d/operationStatuses/08584094598472503244\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6554112-svc/providers/Microsoft.Resources/deployments/cluster-ismqifijm7koe\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6554112-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j6554112-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j6554112-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-15T22:37:03.691Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-15T22:37:05.354Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-15T22:37:06.762Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=bee630fb16cd45492e943a7ab4f37f451fa3266ef665a6d74f650467bd60800f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9396992-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbee630fb16cd45492e943a7ab4f37f451fa3266ef665a6d74f650467bd60800f
time=2026-10-15T22:40:07.453Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9396992-svc/providers/Microsoft.Resources/deployments/bee630fb16cd45492e943a7ab4f37f451fa3266ef665a6d74f650467bd60800f/operationStatuses/08584095030595181402\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9396992-svc/providers/Microsoft.Resources/deployments/cluster-kpgg4tugptwaw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9396992-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j9396992-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j9396992-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-15T20:48:50.569Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-15T20:48:52.496Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-15T20:48:54.249Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8276bf3e2ac415cdaf52404f9c09132aafef64d9fdc663f4e7aca3248fabbce7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3682688-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8276bf3e2ac415cdaf52404f9c09132aafef64d9fdc663f4e7aca3248fabbce7
time=2026-10-15T20:51:24.632Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3682688-svc/providers/Microsoft.Resources/deployments/8276bf3e2ac415cdaf52404f9c09132aafef64d9fdc663f4e7aca3248fabbce7/operationStatuses/08584095095521617911\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3682688-svc/providers/Microsoft.Resources/deployments/cluster-a22genk7v3fe6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3682688-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j3682688-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j3682688-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | e2e | 8 | 1.63%8 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 2 · Oct 13: 1 · Oct 14: 0 · Oct 15: 4 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x1583fe428d80>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qxkjhfl8b89k" to finish creating: GET https://rp.j0669824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb248cdb-708c-4e1a-a191-a9d66edb0204
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb248cdb-708c-4e1a-a191-a9d66edb0204",
"name": "fb248cdb-708c-4e1a-a191-a9d66edb0204",
"status": "Failed",
"startTime": "2026-10-16T06:10:02.689570443Z",
"endTime": "2026-10-16T06:29:04.095727153Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng to exist\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng to exist\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0669824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb248cdb-708c-4e1a-a191-a9d66edb0204
correlation ID: 22bcc657-19c6-431f-9432-afe9f54b1a18
client request ID: 0f3d0f1f-4c8a-4108-a1df-7f5cda2678a1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x1583fe428d80>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-qxkjhfl8b89k" to finish creating: GET https://rp.j0669824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb248cdb-708c-4e1a-a191-a9d66edb0204
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb248cdb-708c-4e1a-a191-a9d66edb0204",
"name": "fb248cdb-708c-4e1a-a191-a9d66edb0204",
"status": "Failed",
"startTime": "2026-10-16T06:10:02.689570443Z",
"endTime": "2026-10-16T06:29:04.095727153Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng to exist\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-dc4ng to exist\nMachine j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-qxkjhfl8b89k--managed/providers/Microsoft.Compute/virtualMachines/j1e2a9k7q9d5m2t-np-1-xwkgq-j9d76 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0669824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb248cdb-708c-4e1a-a191-a9d66edb0204
correlation ID: 22bcc657-19c6-431f-9432-afe9f54b1a18
client request ID: 0f3d0f1f-4c8a-4108-a1df-7f5cda2678a1
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
<*fmt.wrapError | 0x2d528da08240>:
failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-555prpnqd6q9" to finish creating: GET https://rp.j4481792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68716f1-756f-4cdd-a508-f1a64147dcc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68716f1-756f-4cdd-a508-f1a64147dcc7",
"name": "e68716f1-756f-4cdd-a508-f1a64147dcc7",
"status": "Failed",
"startTime": "2026-10-15T22:00:18.084367751Z",
"endTime": "2026-10-15T22:29:20.241927727Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8 to exist\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8 to exist\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4481792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68716f1-756f-4cdd-a508-f1a64147dcc7
correlation ID: 7e1b7dd4-0022-48fb-91be-ab56afbfcaf9
client request ID: eaae09a9-3401-4404-aa69-a9f272cf4e2d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
<*fmt.wrapError | 0x2d528da08240>:
failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-555prpnqd6q9" to finish creating: GET https://rp.j4481792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68716f1-756f-4cdd-a508-f1a64147dcc7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68716f1-756f-4cdd-a508-f1a64147dcc7",
"name": "e68716f1-756f-4cdd-a508-f1a64147dcc7",
"status": "Failed",
"startTime": "2026-10-15T22:00:18.084367751Z",
"endTime": "2026-10-15T22:29:20.241927727Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8 to exist\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-9l6x8 to exist\nMachine u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-555prpnqd6q9--managed/providers/Microsoft.Compute/virtualMachines/u5w8y3l2l4x1v4y-nodepool-128gib-pq9vx-tcn2v to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4481792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68716f1-756f-4cdd-a508-f1a64147dcc7
correlation ID: 7e1b7dd4-0022-48fb-91be-ab56afbfcaf9
client request ID: eaae09a9-3401-4404-aa69-a9f272cf4e2d
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
<*fmt.wrapError | 0x371fb75e40>:
failed to create NodePool swift-workers: failed waiting for nodepool="swift-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-htlwf6tksvkk" to finish creating: GET https://rp.j2353536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b439425f-1c8a-4eb5-8449-8ffb64a115d8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b439425f-1c8a-4eb5-8449-8ffb64a115d8",
"name": "b439425f-1c8a-4eb5-8449-8ffb64a115d8",
"status": "Failed",
"startTime": "2026-10-15T21:32:41.543837723Z",
"endTime": "2026-10-15T22:01:41.868424358Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq to exist\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq to exist\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2353536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b439425f-1c8a-4eb5-8449-8ffb64a115d8
correlation ID: 11b13ff0-4dd4-474f-81b1-b4341cbc7ec0
client request ID: 9a03c5e0-eb37-442c-b9ce-57bebab70a3c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
<*fmt.wrapError | 0x371fb75e40>:
failed to create NodePool swift-workers: failed waiting for nodepool="swift-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-htlwf6tksvkk" to finish creating: GET https://rp.j2353536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b439425f-1c8a-4eb5-8449-8ffb64a115d8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b439425f-1c8a-4eb5-8449-8ffb64a115d8",
"name": "b439425f-1c8a-4eb5-8449-8ffb64a115d8",
"status": "Failed",
"startTime": "2026-10-15T21:32:41.543837723Z",
"endTime": "2026-10-15T22:01:41.868424358Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq to exist\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-4x9jq to exist\nMachine f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-htlwf6tksvkk--managed/providers/Microsoft.Compute/virtualMachines/f0h5o6h1r1j7z7l-swift-workers-wnvr8-t8p9t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2353536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b439425f-1c8a-4eb5-8449-8ffb64a115d8
correlation ID: 11b13ff0-4dd4-474f-81b1-b4341cbc7ec0
client request ID: 9a03c5e0-eb37-442c-b9ce-57bebab70a3c
occurredContributing tests (6)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster "<cluster>" did not complete the SRE-pinned version 4.21.36full failure pattern: cluster "<cluster>" did not complete the SRE-pinned version 4.21.36 Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 7 | 1.43%7 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 5 · Oct 13: 2 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.005s.
cluster "sre-version-pin-z4tqzw" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.005s.
cluster "sre-version-pin-z4tqzw" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.002s.
cluster "sre-version-pin-sjbwq2" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.002s.
cluster "sre-version-pin-sjbwq2" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-k9v4gs" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.003s.
cluster "sre-version-pin-k9v4gs" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36Contributing tests (1)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> firedfull failure pattern: alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | alert | 6 | 1.22%6 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 4 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-14T03:48:00Z Ended: 2026-10-14T04:18:05Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-14T00:46:13Z Ended: 2026-10-14T01:18:14Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T22:09:34Z Ended: 2026-10-13T22:42:14Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units Contributing tests (6)
Affected runs (6) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGridfull failure pattern: ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | provision | 5 | 1.02%5 of 491 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 5 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-16T04:32:52.168Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-16T04:32:53.669Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-16T04:32:54.816Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=52b936b326777547bb7821b194e68c26b111cb76ef998bdad29331fe388cab6a portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3854848%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F52b936b326777547bb7821b194e68c26b111cb76ef998bdad29331fe388cab6a
time=2026-10-16T04:48:28.496Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3854848/providers/Microsoft.Resources/deployments/52b936b326777547bb7821b194e68c26b111cb76ef998bdad29331fe388cab6a/operationStatuses/08584094817114313637\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3854848/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3854848/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j3854848\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-16T03:17:16.536Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-16T03:17:18.501Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-16T03:17:19.523Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=23386a13ae4b9c10e1167d28d0eb0e67c647056f11a0c2bc948bc314d383ed57 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6526208%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F23386a13ae4b9c10e1167d28d0eb0e67c647056f11a0c2bc948bc314d383ed57
time=2026-10-16T03:35:23.849Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6526208/providers/Microsoft.Resources/deployments/23386a13ae4b9c10e1167d28d0eb0e67c647056f11a0c2bc948bc314d383ed57/operationStatuses/08584094862465785031\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6526208/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6526208/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j6526208\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-16T02:17:33.774Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-10-16T02:17:35.249Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-10-16T02:17:36.184Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=11568e45e85c71193ac85f28e42fe8900978b11d9ee87376f97b514ef517dce8 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5876608%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F11568e45e85c71193ac85f28e42fe8900978b11d9ee87376f97b514ef517dce8
time=2026-10-16T02:33:38.292Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5876608/providers/Microsoft.Resources/deployments/11568e45e85c71193ac85f28e42fe8900978b11d9ee87376f97b514ef517dce8/operationStatuses/08584094898298983239\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5876608/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5876608/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j5876608\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] HCPClusterOperatorUnavailable firedfull failure pattern: alert [hcp] HCPClusterOperatorUnavailable fired Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | alert | 5 | 1.02%5 of 491 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 11: 0 · Oct 12: 3 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-16T02:38:55Z Ended: 2026-10-16T03:16:55Z Severity: Sev3 Labels: alertname="HCPClusterOperatorUnavailable", cluster="ci01-j2423808-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tfj30j2k6sdjh9r7vsbj6npes2m40m2-p2r6x1t9q6d6f9l", region="centralus", severity="warning" Description: 1 cluster operator(s) on hosted cluster ocm-arohcpci01-2tfj30j2k6sdjh9r7vsbj6npes2m40m2-p2r6x1t9q6d6f9l (management cluster ci01-j2423808-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-16T02:40:29Z Ended: 2026-10-16T03:15:31Z Severity: Sev3 Labels: alertname="HCPClusterOperatorUnavailable", cluster="ci01-j3528064-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tfj3qrd1lfn90p1i58kr795ndsi2u3g-f4z5z8b0a1u0o2v", region="westus3", severity="warning" Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tfj3qrd1lfn90p1i58kr795ndsi2u3g-f4z5z8b0a1u0o2v (management cluster ci01-j3528064-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-12T21:30:33Z Ended: 2026-10-12T22:07:32Z Severity: Sev3 Labels: alertname="HCPClusterOperatorUnavailable", cluster="ci01-j6043392-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdf8lncsu04rjm64fbreumbi4l47bg3-l5g9b5r3v2p1b0s", region="westus3", severity="warning" Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2tdf8lncsu04rjm64fbreumbi4l47bg3-l5g9b5r3v2p1b0s (management cluster ci01-j6043392-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded. Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
E2E Jobs
0
Success Rate
0.00%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.n/a
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 80.00% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 84.44% | 45 | integration/parallel | Customer should be able to create an HCP cluster then delete it by deleting the customer resource group | |
| 84.44% | 45 | integration/parallel | Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully | |
| 84.44% | 45 | integration/parallel | Customer should be able to update node pool labels and taints |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
No failure patterns available for this environment in the selected window.
Environment: STG
E2E Jobs
12
Success Rate
16.67%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.16.67% (2/12)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 45.45% | 11 | stage/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 45.45% | 11 | stage/parallel | Customer should be able to create a cluster and node pools with aggregated advanced features | |
| 45.45% | 11 | stage/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 45.45% | 11 | stage/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 45.45% | 11 | stage/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | e2e | 6 | 50.00%6 of 12 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | Show trend detailsOct 11: 5 · Oct 12: 4 · Oct 13: 5 · Oct 14: 5 · Oct 15: 5 · Oct 16: 5 · Oct 17: 5 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x4fc6ce77170>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: 09054a39-11cf-4b09-a485-31e6d3a5caaf
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x4fc6ce77170>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: 09054a39-11cf-4b09-a485-31e6d3a5caaf
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
<*exported.ResponseError | 0x2195bd58cde0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
correlation ID: 4e9f2a97-920c-4952-a752-7a5aa59c6506
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
<*exported.ResponseError | 0x2195bd58cde0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
correlation ID: 4e9f2a97-920c-4952-a752-7a5aa59c6506
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x8e1dd31d290>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: bf0c2965-cddd-42e6-9d62-d3767563c18a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x8e1dd31d290>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: bf0c2965-cddd-42e6-9d62-d3767563c18a
client request ID: unavailable
occurredContributing tests (5)
Affected runs (6)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 33.33%4 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 1 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x5d768dbf900>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x5d768dbf900>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x39d9d422b5e0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x39d9d422b5e0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x284d46b12080>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x284d46b12080>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (2)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHyperShiftIngressCertificate timed out after 10m0full failure pattern: VerifyHyperShiftIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 33.33%4 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 1 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1526b2d700a0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70080>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70060>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1526b2d700a0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70080>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70060>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x21f4804320>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f48042e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f4804280>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x21f4804320>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f48042e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f4804280>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x27807b514a20>:
VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b5149e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b514980>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x27807b514a20>:
VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b5149e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b514980>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (2)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
external auth <external-auth>.Status was nilfull failure pattern: external auth <external-auth>.Status was nil Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 33.33%4 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 1 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.000s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.000s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.604s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.604s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.002s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.002s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilContributing tests (2)
Affected runs (4)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 3 | 25.00%3 of 12 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 11: 2 · Oct 12: 5 · Oct 13: 1 · Oct 14: 3 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-snb95l" at install version 4.22.0-0.nightly-multi-2026-10-14-193612
Unexpected error:
<*fmt.wrapError | 0x1f800f0eba0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-snb95l: failed waiting for cluster="cp-ystream-upgrade-5-0-snb95l" in resourcegroup="rg-cp-ystream-upgrade-5-0-9pm7qw6t4kxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716",
"name": "5b5b73d5-ad43-4248-a379-3589ba793716",
"status": "Failed",
"startTime": "2026-10-14T23:28:48.207254546Z",
"endTime": "2026-10-14T23:57:58.000803929Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
correlation ID: 8af90513-a16b-4576-aebf-be0202178446
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-snb95l" at install version 4.22.0-0.nightly-multi-2026-10-14-193612
Unexpected error:
<*fmt.wrapError | 0x1f800f0eba0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-snb95l: failed waiting for cluster="cp-ystream-upgrade-5-0-snb95l" in resourcegroup="rg-cp-ystream-upgrade-5-0-9pm7qw6t4kxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716",
"name": "5b5b73d5-ad43-4248-a379-3589ba793716",
"status": "Failed",
"startTime": "2026-10-14T23:28:48.207254546Z",
"endTime": "2026-10-14T23:57:58.000803929Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
correlation ID: 8af90513-a16b-4576-aebf-be0202178446
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-4bfb22rwmqzl/cluster-nightly-4-22-8zxbz4 should provision
Unexpected error:
<*fmt.wrapError | 0x363444c03360>:
failed to create HCP cluster cluster-nightly-4-22-8zxbz4: failed waiting for cluster="cluster-nightly-4-22-8zxbz4" in resourcegroup="rg-nightly-4-22-4bfb22rwmqzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"name": "4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"status": "Failed",
"startTime": "2026-10-14T23:25:15.190269818Z",
"endTime": "2026-10-14T23:54:17.885160466Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
correlation ID: eee8c816-0fc0-4cfa-8c09-a91c4755ee62
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-4bfb22rwmqzl/cluster-nightly-4-22-8zxbz4 should provision
Unexpected error:
<*fmt.wrapError | 0x363444c03360>:
failed to create HCP cluster cluster-nightly-4-22-8zxbz4: failed waiting for cluster="cluster-nightly-4-22-8zxbz4" in resourcegroup="rg-nightly-4-22-4bfb22rwmqzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"name": "4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"status": "Failed",
"startTime": "2026-10-14T23:25:15.190269818Z",
"endTime": "2026-10-14T23:54:17.885160466Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
correlation ID: eee8c816-0fc0-4cfa-8c09-a91c4755ee62
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-7c4xdncsltnp/cluster-nightly-4-21-7xdrt5 should provision
Unexpected error:
<*fmt.wrapError | 0xc38fe936560>:
failed to create HCP cluster cluster-nightly-4-21-7xdrt5: failed waiting for cluster="cluster-nightly-4-21-7xdrt5" in resourcegroup="rg-nightly-4-21-7c4xdncsltnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"name": "bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"status": "Failed",
"startTime": "2026-10-14T23:25:05.12965498Z",
"endTime": "2026-10-14T23:54:07.890200924Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
correlation ID: 0f756d77-a36e-4289-82ab-c5ef8b73feb4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-7c4xdncsltnp/cluster-nightly-4-21-7xdrt5 should provision
Unexpected error:
<*fmt.wrapError | 0xc38fe936560>:
failed to create HCP cluster cluster-nightly-4-21-7xdrt5: failed waiting for cluster="cluster-nightly-4-21-7xdrt5" in resourcegroup="rg-nightly-4-21-7c4xdncsltnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"name": "bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"status": "Failed",
"startTime": "2026-10-14T23:25:05.12965498Z",
"endTime": "2026-10-14T23:54:07.890200924Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
correlation ID: 0f756d77-a36e-4289-82ab-c5ef8b73feb4
client request ID: unavailable
occurredContributing tests (7)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 2 | 16.67%2 of 12 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 11: 1 · Oct 12: 1 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-qfmkcj" at install version 5.0.0-0.nightly-multi-2026-10-14-152123
Unexpected error:
<*fmt.wrapError | 0x33329457b0a0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-qfmkcj: failed waiting for cluster="cp-ystream-upgrade-5-1-qfmkcj" in resourcegroup="rg-cp-ystream-upgrade-5-1-667r4hptzmjc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0",
"name": "b88eb811-d667-4408-bb78-6c86786b44f0",
"status": "Failed",
"startTime": "2026-10-14T23:29:34.061824976Z",
"endTime": "2026-10-14T23:58:37.90403656Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
correlation ID: 8de1d04b-8264-4a06-8468-c991e9773a81
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-qfmkcj" at install version 5.0.0-0.nightly-multi-2026-10-14-152123
Unexpected error:
<*fmt.wrapError | 0x33329457b0a0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-qfmkcj: failed waiting for cluster="cp-ystream-upgrade-5-1-qfmkcj" in resourcegroup="rg-cp-ystream-upgrade-5-1-667r4hptzmjc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0",
"name": "b88eb811-d667-4408-bb78-6c86786b44f0",
"status": "Failed",
"startTime": "2026-10-14T23:29:34.061824976Z",
"endTime": "2026-10-14T23:58:37.90403656Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
correlation ID: 8de1d04b-8264-4a06-8468-c991e9773a81
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
<*fmt.wrapError | 0x2d73d9d10e00>:
failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
"name": "41bf70ba-7344-4c62-9364-b6165697473d",
"status": "Failed",
"startTime": "2026-10-12T23:27:02.793421826Z",
"endTime": "2026-10-12T23:56:08.063137348Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
<*fmt.wrapError | 0x2d73d9d10e00>:
failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
"name": "41bf70ba-7344-4c62-9364-b6165697473d",
"status": "Failed",
"startTime": "2026-10-12T23:27:02.793421826Z",
"endTime": "2026-10-12T23:56:08.063137348Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 16.67%2 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x2e2df3fce3a0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-cxh9xv97rj9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be",
"name": "0195b61d-2798-45a3-a546-b2e316a8e4be",
"status": "Failed",
"startTime": "2026-10-14T03:10:09.167253269Z",
"endTime": "2026-10-14T03:39:12.28722327Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
correlation ID: 6295da04-1f90-4641-a016-0502957f2d44
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x2e2df3fce3a0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-cxh9xv97rj9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be",
"name": "0195b61d-2798-45a3-a546-b2e316a8e4be",
"status": "Failed",
"startTime": "2026-10-14T03:10:09.167253269Z",
"endTime": "2026-10-14T03:39:12.28722327Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
correlation ID: 6295da04-1f90-4641-a016-0502957f2d44
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x371cbc791620>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
"name": "7da88189-d1ef-4299-b869-95f7299be069",
"status": "Failed",
"startTime": "2026-10-12T23:47:30.718672913Z",
"endTime": "2026-10-13T00:16:35.706367535Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x371cbc791620>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
"name": "7da88189-d1ef-4299-b869-95f7299be069",
"status": "Failed",
"startTime": "2026-10-12T23:47:30.718672913Z",
"endTime": "2026-10-13T00:16:35.706367535Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...full failure pattern: ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 11 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
<*fmt.wrapError | 0x15168f06cae0>:
failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
"name": "8479abf5-39da-412b-bb04-87374bf3b757",
"status": "Failed",
"startTime": "2026-10-13T00:45:19.685007768Z",
"endTime": "2026-10-13T01:14:27.351105107Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
<*fmt.wrapError | 0x15168f06cae0>:
failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
"name": "8479abf5-39da-412b-bb04-87374bf3b757",
"status": "Failed",
"startTime": "2026-10-13T00:45:19.685007768Z",
"endTime": "2026-10-13T01:14:27.351105107Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x3152b3c1a500>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"status": "Failed",
"startTime": "2026-10-13T00:45:30.036006045Z",
"endTime": "2026-10-13T01:14:37.44566298Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x3152b3c1a500>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"status": "Failed",
"startTime": "2026-10-13T00:45:30.036006045Z",
"endTime": "2026-10-13T01:14:37.44566298Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x15a1e46902c0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
"name": "8f006a7a-53bf-4566-93d0-8202c11da346",
"status": "Failed",
"startTime": "2026-10-13T00:52:36.6753625Z",
"endTime": "2026-10-13T01:21:37.350837076Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x15a1e46902c0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
"name": "8f006a7a-53bf-4566-93d0-8202c11da346",
"status": "Failed",
"startTime": "2026-10-13T00:52:36.6753625Z",
"endTime": "2026-10-13T01:21:37.350837076Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
client request ID: unavailable
occurredContributing tests (11)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenSh...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 10 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x1854e4898c30>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x1854e4898c30>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
<*exported.ResponseError | 0x13a151ae2450>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
<*exported.ResponseError | 0x13a151ae2450>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
<*exported.ResponseError | 0x1d06db38a6c0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
<*exported.ResponseError | 0x1d06db38a6c0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenS...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 9 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x31faef804060>:
failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x31faef804060>:
failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
<*fmt.wrapError | 0x258acfd8b940>:
failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
<*fmt.wrapError | 0x258acfd8b940>:
failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
<*fmt.wrapError | 0x1b553dfaa40>:
failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
<*fmt.wrapError | 0x1b553dfaa40>:
failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
client request ID: unavailable
occurredContributing tests (9)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpe...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 3 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
<*exported.ResponseError | 0x2299302636e0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
<*exported.ResponseError | 0x2299302636e0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
<*exported.ResponseError | 0x33ccd41339b0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
<*exported.ResponseError | 0x33ccd41339b0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
<*exported.ResponseError | 0x1206c2abaf90>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
<*exported.ResponseError | 0x1206c2abaf90>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x95f8488fc80>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-n6c4c4fd2ssk" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-n6c4c4fd2ssk\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x95f84458c30>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x95f8488fc80>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-n6c4c4fd2ssk" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-n6c4c4fd2ssk\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x95f84458c30>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
<*fmt.wrapError | 0xc4ca110a7e0>:
failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"status": "Failed",
"startTime": "2026-10-12T23:27:12.849511271Z",
"endTime": "2026-10-12T23:56:15.713757928Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
<*fmt.wrapError | 0xc4ca110a7e0>:
failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"status": "Failed",
"startTime": "2026-10-12T23:27:12.849511271Z",
"endTime": "2026-10-12T23:56:15.713757928Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-zxqcgtfv7nf5/cluster-nightly-5-1-2dz545 should provision
Unexpected error:
<*fmt.wrapError | 0x3ec63f648620>:
failed to create HCP cluster cluster-nightly-5-1-2dz545: failed waiting for cluster="cluster-nightly-5-1-2dz545" in resourcegroup="rg-nightly-5-1-zxqcgtfv7nf5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"name": "c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"status": "Failed",
"startTime": "2026-10-14T23:26:17.571795133Z",
"endTime": "2026-10-14T23:55:18.071165518Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, catalog-operator, packageserver, openshift-controller-manager, dns-operator, openshift-route-controller-manager, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
correlation ID: b45fa2ae-f53a-4948-b0fb-190aed3be0b8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-zxqcgtfv7nf5/cluster-nightly-5-1-2dz545 should provision
Unexpected error:
<*fmt.wrapError | 0x3ec63f648620>:
failed to create HCP cluster cluster-nightly-5-1-2dz545: failed waiting for cluster="cluster-nightly-5-1-2dz545" in resourcegroup="rg-nightly-5-1-zxqcgtfv7nf5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"name": "c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"status": "Failed",
"startTime": "2026-10-14T23:26:17.571795133Z",
"endTime": "2026-10-14T23:55:18.071165518Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, catalog-operator, packageserver, openshift-controller-manager, dns-operator, openshift-route-controller-manager, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
correlation ID: b45fa2ae-f53a-4948-b0fb-190aed3be0b8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-8qt7kcrvhtlb/cluster-nightly-5-0-6wsq4v should provision
Unexpected error:
<*fmt.wrapError | 0x30f3685a760>:
failed to create HCP cluster cluster-nightly-5-0-6wsq4v: failed waiting for cluster="cluster-nightly-5-0-6wsq4v" in resourcegroup="rg-nightly-5-0-8qt7kcrvhtlb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270",
"name": "70ee1e74-bffb-4dec-89fb-282756ca2270",
"status": "Failed",
"startTime": "2026-10-14T23:26:07.782117382Z",
"endTime": "2026-10-14T23:55:07.903819031Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-0.nightly-multi-2026-10-14-152123 is Partial (want Completed), started 20m19s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
correlation ID: e153794f-fb7c-4e3b-9441-a5dadf7b0225
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-8qt7kcrvhtlb/cluster-nightly-5-0-6wsq4v should provision
Unexpected error:
<*fmt.wrapError | 0x30f3685a760>:
failed to create HCP cluster cluster-nightly-5-0-6wsq4v: failed waiting for cluster="cluster-nightly-5-0-6wsq4v" in resourcegroup="rg-nightly-5-0-8qt7kcrvhtlb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270",
"name": "70ee1e74-bffb-4dec-89fb-282756ca2270",
"status": "Failed",
"startTime": "2026-10-14T23:26:07.782117382Z",
"endTime": "2026-10-14T23:55:07.903819031Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-0.nightly-multi-2026-10-14-152123 is Partial (want Completed), started 20m19s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
correlation ID: e153794f-fb7c-4e3b-9441-a5dadf7b0225
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 8.33%1 of 12 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x3d800eb17200>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3d800f297260>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3d800f297240>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3d800e191e00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3d800e204000>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3d800eb05ee0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3d800e0fab40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x3d800eb17200>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3d800f297260>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3d800f297240>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3d800e191e00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3d800e204000>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3d800eb05ee0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3d800e0fab40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
E2E Jobs
37
Success Rate
43.24%
E2E success
Among runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step.45.71% (16/35)
Daily Run Outcomes
Successful runs
Provision failures
E2E failures
Other failures
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 36.36% | 11 | prod/parallel | Customer should be able to create a cluster with an external auth config and get the external auth config | |
| 45.45% | 11 | prod/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 45.45% | 11 | prod/parallel | Customer should be able to create a cluster and node pools with aggregated advanced features | |
| 45.45% | 11 | prod/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 45.45% | 11 | prod/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 8 days | e2e | 7 | 18.92%7 of 37 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 8 days | Show trend detailsOct 11: 4 · Oct 12: 4 · Oct 13: 5 · Oct 14: 5 · Oct 15: 5 · Oct 16: 5 · Oct 17: 5 · Oct 18: 5 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
<*exported.ResponseError | 0x2131411bbc0>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-7vlft5fbxfj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-7vlft5fbxfj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
correlation ID: 082a1be7-a852-4fa9-86e9-0d81abc827d5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
<*exported.ResponseError | 0x2131411bbc0>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-7vlft5fbxfj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-7vlft5fbxfj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
correlation ID: 082a1be7-a852-4fa9-86e9-0d81abc827d5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x2b96e1fd0840>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-s7wtmrpb4ccm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-s7wtmrpb4ccm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: ef97d78e-5044-4679-8aa8-0997eb8f50e0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x2b96e1fd0840>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-s7wtmrpb4ccm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-s7wtmrpb4ccm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: ef97d78e-5044-4679-8aa8-0997eb8f50e0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x2a07de41b7d0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tkchjbmblqjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tkchjbmblqjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: fdd287eb-94a9-42d8-b989-4e063f2b89e8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x2a07de41b7d0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tkchjbmblqjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tkchjbmblqjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: fdd287eb-94a9-42d8-b989-4e063f2b89e8
client request ID: unavailable
occurredContributing tests (5)
Affected runs (7)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 5 | 13.51%5 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 2 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x117f03d50190>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x117f03d50190>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x22d17affc460>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x22d17affc460>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x1b882be0fa60>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:509]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x1b882be0fa60>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (2)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHyperShiftIngressCertificate timed out after 10m0full failure pattern: VerifyHyperShiftIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 5 | 13.51%5 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 2 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0xa8f3937e880>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e840>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e7c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0xa8f3937e880>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e840>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e7c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1025ee5cc400>:
VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1025ee5cc400>:
VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x122fed28cd60>:
VerifyHyperShiftIngressCertificate timed out after 10m0.004s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.004s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x122fed28cd20>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x122fed28cce0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x122fed28cd60>:
VerifyHyperShiftIngressCertificate timed out after 10m0.004s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.004s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x122fed28cd20>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x122fed28cce0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (2)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
external auth <external-auth>.Status was nilfull failure pattern: external auth <external-auth>.Status was nil Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 5 | 13.51%5 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 2 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.004s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.004s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.003s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.003s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.001s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.001s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilContributing tests (2)
Affected runs (5)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 8.11%3 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 2 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x1d3b45812a50>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-5lx5864xtdrb" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-5lx5864xtdrb\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1d3b4591ca00>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x1d3b45812a50>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-5lx5864xtdrb" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-5lx5864xtdrb\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1d3b4591ca00>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x2be35a9dc6c0>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-4swfh9bhdjlh" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-4swfh9bhdjlh\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2be35bcc7190>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x2be35a9dc6c0>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-4swfh9bhdjlh" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-4swfh9bhdjlh\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2be35bcc7190>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x130faa8f320>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-bkm7lwv87ztr" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-bkm7lwv87ztr\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x130f942f8e0>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x130faa8f320>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-bkm7lwv87ztr" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-bkm7lwv87ztr\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x130f942f8e0>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 8.11%3 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 2 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x25ccdbb84fe0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-fwqz99tj9vqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f",
"name": "3b0e9b20-034d-4163-98db-b3ac4dede16f",
"status": "Failed",
"startTime": "2026-10-15T07:00:37.486714177Z",
"endTime": "2026-10-15T07:29:37.711665502Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
correlation ID: 0ad357fe-5f92-4836-9361-6bb0422c841f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x25ccdbb84fe0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-fwqz99tj9vqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f",
"name": "3b0e9b20-034d-4163-98db-b3ac4dede16f",
"status": "Failed",
"startTime": "2026-10-15T07:00:37.486714177Z",
"endTime": "2026-10-15T07:29:37.711665502Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
correlation ID: 0ad357fe-5f92-4836-9361-6bb0422c841f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x10ea921035c0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-4426rkkk84lt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"name": "cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"status": "Failed",
"startTime": "2026-10-15T04:26:27.342644977Z",
"endTime": "2026-10-15T04:55:33.710462341Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
correlation ID: 442271c1-98e4-4b63-9254-ce9229dd3320
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x10ea921035c0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-4426rkkk84lt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"name": "cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"status": "Failed",
"startTime": "2026-10-15T04:26:27.342644977Z",
"endTime": "2026-10-15T04:55:33.710462341Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
correlation ID: 442271c1-98e4-4b63-9254-ce9229dd3320
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x151c97ead140>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-p86tvztrm7hd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b",
"name": "01aaf869-754a-4412-a8d0-48b576bc7f7b",
"status": "Failed",
"startTime": "2026-10-12T01:36:23.272806003Z",
"endTime": "2026-10-12T02:05:32.513113167Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
correlation ID: e3c8027d-936a-4cfa-802c-eac5497d3369
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x151c97ead140>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-p86tvztrm7hd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b",
"name": "01aaf869-754a-4412-a8d0-48b576bc7f7b",
"status": "Failed",
"startTime": "2026-10-12T01:36:23.272806003Z",
"endTime": "2026-10-12T02:05:32.513113167Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-p86tvztrm7hd--managed/providers/Microsoft.Compute/virtualMachines/u6e7j5d8a2b3s1a-cordon-np-bp4qm-n5n47 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/01aaf869-754a-4412-a8d0-48b576bc7f7b
correlation ID: e3c8027d-936a-4cfa-802c-eac5497d3369
client request ID: unavailable
occurredContributing tests (2)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 5.41%2 of 37 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:106]: failed to verify HCP cluster "cx-rg-hcp-cluster" is viable
Unexpected error:
<*errors.joinError | 0x9c519e26018>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x9c519869940>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x9c519869920>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x9c5196cebe0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x9c51a1232c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x9c51a1232a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x9c519493900>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:106]: failed to verify HCP cluster "cx-rg-hcp-cluster" is viable
Unexpected error:
<*errors.joinError | 0x9c519e26018>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x9c519869940>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x9c519869920>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x9c5196cebe0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x9c51a1232c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x9c51a1232a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x9c519493900>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:175]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x2d289f100030>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d289e7b3500>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d289e7b34e0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d289f0dae60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2d289e6a6040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2d289e6a6020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2d289e168dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:175]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x2d289f100030>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d289e7b3500>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d289e7b34e0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d289f0dae60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2d289e6a6040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2d289e6a6020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2d289e168dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 5.41%2 of 37 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x3a1769a66d00>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4m8h2dg477s7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705",
"name": "e7a3692b-5901-4199-ae87-9018f084c705",
"status": "Failed",
"startTime": "2026-10-15T06:48:50.851858954Z",
"endTime": "2026-10-15T07:17:54.261659106Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
correlation ID: 23578d08-9577-490e-b796-4ce4da9b9b97
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x3a1769a66d00>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-4m8h2dg477s7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705",
"name": "e7a3692b-5901-4199-ae87-9018f084c705",
"status": "Failed",
"startTime": "2026-10-15T06:48:50.851858954Z",
"endTime": "2026-10-15T07:17:54.261659106Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e7a3692b-5901-4199-ae87-9018f084c705
correlation ID: 23578d08-9577-490e-b796-4ce4da9b9b97
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ptqdgd" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x21d67b55ea20>:
failed to create HCP cluster cp-ystream-upgrade-5-0-ptqdgd: failed waiting for cluster="cp-ystream-upgrade-5-0-ptqdgd" in resourcegroup="rg-cp-ystream-upgrade-5-0-dznstqd64m54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044",
"name": "9d14066e-1629-48f8-aa68-f556c4fd7044",
"status": "Failed",
"startTime": "2026-10-12T07:51:24.545231929Z",
"endTime": "2026-10-12T08:20:29.935309867Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
correlation ID: 7a60ad0c-37f6-45ef-98dd-efdfad687d6c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ptqdgd" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x21d67b55ea20>:
failed to create HCP cluster cp-ystream-upgrade-5-0-ptqdgd: failed waiting for cluster="cp-ystream-upgrade-5-0-ptqdgd" in resourcegroup="rg-cp-ystream-upgrade-5-0-dznstqd64m54" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044",
"name": "9d14066e-1629-48f8-aa68-f556c4fd7044",
"status": "Failed",
"startTime": "2026-10-12T07:51:24.545231929Z",
"endTime": "2026-10-12T08:20:29.935309867Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/9d14066e-1629-48f8-aa68-f556c4fd7044
correlation ID: 7a60ad0c-37f6-45ef-98dd-efdfad687d6c
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 5.41%2 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 2 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x3d468a7c3d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdw8m7lp2fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"name": "3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"status": "Failed",
"startTime": "2026-10-15T23:18:55.172688796Z",
"endTime": "2026-10-15T23:47:59.507731058Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
correlation ID: df77bc05-17d5-4c4c-bad3-0bd2a5dbd8cc
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x3d468a7c3d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdw8m7lp2fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"name": "3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"status": "Failed",
"startTime": "2026-10-15T23:18:55.172688796Z",
"endTime": "2026-10-15T23:47:59.507731058Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
correlation ID: df77bc05-17d5-4c4c-bad3-0bd2a5dbd8cc
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x7be4d399a60>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-qqvhz8j7w227" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233",
"name": "8506345f-be03-477f-be2f-7b05e552a233",
"status": "Failed",
"startTime": "2026-10-15T13:54:56.565235769Z",
"endTime": "2026-10-15T14:23:56.214949667Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
correlation ID: 70e8c338-b9fd-439c-b234-059edc54dfd9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x7be4d399a60>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-qqvhz8j7w227" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233",
"name": "8506345f-be03-477f-be2f-7b05e552a233",
"status": "Failed",
"startTime": "2026-10-15T13:54:56.565235769Z",
"endTime": "2026-10-15T14:23:56.214949667Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-t5xfb to exist\nMachine d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qqvhz8j7w227--managed/providers/Microsoft.Compute/virtualMachines/d0e4s8o7w0o1m8p-cordon-np-krf75-vxfkg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8506345f-be03-477f-be2f-7b05e552a233
correlation ID: 70e8c338-b9fd-439c-b234-059edc54dfd9
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x16bfc8866960>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-lrl2sfwlg6t6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"name": "bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"status": "Failed",
"startTime": "2026-10-16T23:12:01.466039152Z",
"endTime": "2026-10-16T23:41:09.816691023Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
correlation ID: 696a9369-e43c-4253-91d0-e6c9182748ca
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x16bfc8866960>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-lrl2sfwlg6t6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"name": "bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"status": "Failed",
"startTime": "2026-10-16T23:12:01.466039152Z",
"endTime": "2026-10-16T23:41:09.816691023Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
correlation ID: 696a9369-e43c-4253-91d0-e6c9182748ca
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-q7jjp7" with version 4.21.35 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x298b49702ce0>:
failed to create HCP cluster cluster-zstream-4-21-q7jjp7: failed waiting for cluster="cluster-zstream-4-21-q7jjp7" in resourcegroup="rg-zstream-upgrade-4-21-jd6h7xk8btv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
"name": "c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
"status": "Failed",
"startTime": "2026-10-12T14:02:09.836072198Z",
"endTime": "2026-10-12T14:31:10.061357256Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, packageserver, cluster-network-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
correlation ID: 414509d7-91c9-4911-b025-fd41c0e067b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-q7jjp7" with version 4.21.35 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x298b49702ce0>:
failed to create HCP cluster cluster-zstream-4-21-q7jjp7: failed waiting for cluster="cluster-zstream-4-21-q7jjp7" in resourcegroup="rg-zstream-upgrade-4-21-jd6h7xk8btv9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
"name": "c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8",
"status": "Failed",
"startTime": "2026-10-12T14:02:09.836072198Z",
"endTime": "2026-10-12T14:31:10.061357256Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, packageserver, cluster-network-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, redhat-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, dns-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c104c66f-5dc3-4f93-b1cf-b5a10eca8fb8
correlation ID: 414509d7-91c9-4911-b025-fd41c0e067b1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x38da2adfeb00>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-kbw6v2d2bhz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095",
"name": "4899c274-4471-4ab9-871b-22368aa84095",
"status": "Failed",
"startTime": "2026-10-12T00:35:15.167750325Z",
"endTime": "2026-10-12T01:04:20.403308122Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-fk5wl-zdgdp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-fk5wl-zdgdp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
correlation ID: 79b16636-2655-4af8-9ac0-f8e4eda0ad35
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x38da2adfeb00>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-kbw6v2d2bhz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095",
"name": "4899c274-4471-4ab9-871b-22368aa84095",
"status": "Failed",
"startTime": "2026-10-12T00:35:15.167750325Z",
"endTime": "2026-10-12T01:04:20.403308122Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-fk5wl-zdgdp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-fk5wl-zdgdp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-kbw6v2d2bhz5--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-fk5wl-zdgdp to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4899c274-4471-4ab9-871b-22368aa84095
correlation ID: 79b16636-2655-4af8-9ac0-f8e4eda0ad35
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x13ac3c718210>:
resource group "rg-nightly-4-22-mf542htcmkph": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-22-8ktwxt" in resourcegroup="rg-nightly-4-22-mf542htcmkph" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cf27134-aa80-4f7f-83e4-ed95ed8adec3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cf27134-aa80-4f7f-83e4-ed95ed8adec3",
"name": "2cf27134-aa80-4f7f-83e4-ed95ed8adec3",
"status": "Failed",
"startTime": "2026-10-14T23:46:31.678539487Z",
"endTime": "2026-10-15T00:10:39.404101975Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tes3uu6rr6ujm331pj4rtffuvukefm7 still exists (deletion dispatched at 2026-10-14T23:46:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cf27134-aa80-4f7f-83e4-ed95ed8adec3
correlation ID: 485192e6-34cc-499a-b81c-b88c264c35e8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x13ac3c718210>:
resource group "rg-nightly-4-22-mf542htcmkph": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-22-8ktwxt" in resourcegroup="rg-nightly-4-22-mf542htcmkph" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cf27134-aa80-4f7f-83e4-ed95ed8adec3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cf27134-aa80-4f7f-83e4-ed95ed8adec3",
"name": "2cf27134-aa80-4f7f-83e4-ed95ed8adec3",
"status": "Failed",
"startTime": "2026-10-14T23:46:31.678539487Z",
"endTime": "2026-10-15T00:10:39.404101975Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tes3uu6rr6ujm331pj4rtffuvukefm7 still exists (deletion dispatched at 2026-10-14T23:46:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2cf27134-aa80-4f7f-83e4-ed95ed8adec3
correlation ID: 485192e6-34cc-499a-b81c-b88c264c35e8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining ...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
<*errors.joinError | 0xc79feec4060>:
resource group "np-autoscaling-xtg8j7kg4fwz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-xtg8j7kg4fwz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854",
"name": "6af3ef7c-4974-4b33-bebb-ed53f86cb854",
"status": "Failed",
"startTime": "2026-10-12T02:47:19.509292457Z",
"endTime": "2026-10-12T03:11:19.252244159Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
<*errors.joinError | 0xc79feec4060>:
resource group "np-autoscaling-xtg8j7kg4fwz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-xtg8j7kg4fwz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6af3ef7c-4974-4b33-bebb-ed53f86cb854",
"name": "6af3ef7c-4974-4b33-bebb-ed53f86cb854",
"status": "Failed",
"startTime": "2026-10-12T02:47:19.509292457Z",
"endTime": "2026-10-12T03:11:19.252244159Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 1 · Oct 18: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x26a811ac2138>:
resource group "rg-nightly-4-20-t74cxvx2rdnz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-cb8gb2" in resourcegroup="rg-nightly-4-20-t74cxvx2rdnz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef",
"name": "51202163-2b8d-4c4b-abd9-e60c454750ef",
"status": "Failed",
"startTime": "2026-10-17T23:43:19.794851177Z",
"endTime": "2026-10-18T00:07:19.354701033Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
correlation ID: d4f725b2-2286-488f-91cf-637285920d3b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x26a811ac2138>:
resource group "rg-nightly-4-20-t74cxvx2rdnz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-cb8gb2" in resourcegroup="rg-nightly-4-20-t74cxvx2rdnz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef",
"name": "51202163-2b8d-4c4b-abd9-e60c454750ef",
"status": "Failed",
"startTime": "2026-10-17T23:43:19.794851177Z",
"endTime": "2026-10-18T00:07:19.354701033Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
correlation ID: d4f725b2-2286-488f-91cf-637285920d3b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
There is no failure as the matcher passed to Eventually succeeded on its most recent iterationfull failure pattern: There is no failure as the matcher passed to Eventually succeeded on its most recent iteration Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.000s. delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued There is no failure as the matcher passed to Eventually succeeded on its most recent iteration fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.000s. delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued There is no failure as the matcher passed to Eventually succeeded on its most recent iteration Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||
waiting for a presented DNS-01 challenge for *full failure pattern: waiting for a presented DNS-01 challenge for * Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.70%1 of 37 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.003s.
delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued
Expected success, but got an error:
<*errors.errorString | 0x2ce43d7a8400>:
waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io
{
s: "waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.003s.
delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued
Expected success, but got an error:
<*errors.errorString | 0x2ce43d7a8400>:
waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io
{
s: "waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.lewk.australiaeast.aroapp-hcp.io",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||