Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 77 | 68.83% (off track) | -26.17pp | +16.20pp | 89.61% (69/77) | +5.11pp | 76.81% (53/69) | +12.59pp |
| INT | 95% - All E2E job runs | 26 | 65.38% (off track) | -29.62pp | +0.87pp | n/a | n/a | 85.00% (17/20) | -10.24pp |
| STG | 95% - All E2E job runs | 14 | 85.71% (off track) | -9.29pp | +8.79pp | n/a | n/a | 85.71% (12/14) | +5.71pp |
| PROD | 95% - All E2E job runs | 42 | 76.19% (off track) | -18.81pp | +34.81pp | n/a | n/a | 76.19% (32/42) | +34.09pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 7 | 3.65%7 of 192 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsNov 10: 2 · Nov 11: 1 · Nov 12: 2 · Nov 13: 4 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-14T00:10:33Z Ended: 2026-11-14T00:16:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5892608-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2l7k5ong7lhsdb3rmd7e6f9npoppbg-cluster", pod="openshift-apiserver-65cdb567cb-vrzcv", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2u2l7k5ong7lhsdb3rmd7e6f9npoppbg-cluster/openshift-apiserver-65cdb567cb-vrzcv has been in a non-ready state for longer than 5 minutes. alert fired 32 time(s) (14 unknown, 18 known) Firing 1: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster", pod="kube-apiserver-578576bffb-f6whx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster/kube-apiserver-578576bffb-f6whx has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:41:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:40:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="router-c76569fc9-6cfvc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/router-c76569fc9-6cfvc has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-11-13T22:34:33Z Ended: 2026-11-13T22:41:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpes06vummjujuaun3cf6766c5d4-cluster", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpes06vummjujuaun3cf6766c5d4-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-11-13T22:41:33Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster", pod="etcd-1", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jv5ebnlh0dgm3at8h4he3ssjb88cj-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvd6vuver3peg6br0ldulhn5em1d2-cluster", pod="kube-state-metrics-hcp-867779fc97-5kqgw", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvd6vuver3peg6br0ldulhn5em1d2-cluster/kube-state-metrics-hcp-867779fc97-5kqgw has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:49:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster", pod="openshift-apiserver-548798444f-fkvbc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster/openshift-apiserver-548798444f-fkvbc has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster", pod="openshift-apiserver-7db68f8b9c-ztk8f", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juu5hqds93v8voe4hg2kle1ucc5cq-cluster/openshift-apiserver-7db68f8b9c-ztk8f has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2k00np1fd7kopst9snbjsh5ghmkl25-cluster", pod="kube-state-metrics-hcp-867779fc97-4rb5d", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2k00np1fd7kopst9snbjsh5ghmkl25-cluster/kube-state-metrics-hcp-867779fc97-4rb5d has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster", pod="kube-controller-manager-7b9955c77b-gt9w8", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juqtcaggrvlnh46sqdup5uq7i95gu-cluster/kube-controller-manager-7b9955c77b-gt9w8 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2k00tuj5acf49s1c11mi5h9ldau5ps-cluster", pod="openshift-apiserver-69b9c969cf-2bfxx", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2k00tuj5acf49s1c11mi5h9ldau5ps-cluster/openshift-apiserver-69b9c969cf-2bfxx has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-11-13T22:42:37Z Ended: 2026-11-13T22:48:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster", pod="kube-state-metrics-hcp-867779fc97-nnqft", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2jvpfmiie7puvbcdljnjqbnlvaenik-cluster/kube-state-metrics-hcp-867779fc97-nnqft has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-11-13T22:46:32Z Ended: 2026-11-13T22:52:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2juvn997fddu0ckuf17ouet8eoim9t-cluster", pod="openshift-apiserver-f95d96f84-p58r8", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2u2juvn997fddu0ckuf17ouet8eoim9t-cluster/openshift-apiserver-f95d96f84-p58r8 has been in a non-ready state for longer than 5 minutes. alert fired 14 time(s) (1 unknown, 13 known) Firing 1: State: Resolved Started: 2026-11-13T13:12:50Z Ended: 2026-11-13T13:28:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5993600-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster", pod="etcd-1", region="canadacentral", severity="warning" Description: Pod ocm-arohcpci01-2u2bo6a974okd2bibid30qrn7tb9sok0-cluster/etcd-1 has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (7) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 5 | 2.60%5 of 192 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 2 · Nov 14: 2 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-15T00:38:18Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8647040-mgmt-1", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-ntrqw", region="centralus", severity="warning" Description: Pod swift-recorder/swift-recorder-ntrqw has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-14T15:20:53Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j0519808-svc", component="kubernetes-infrastructure", namespace="aks-command", pod="command-1bae86b83ff941b29f7e8e234ffad7cd", region="westus3", severity="warning" Description: Pod aks-command/command-1bae86b83ff941b29f7e8e234ffad7cd has been in a non-ready state for longer than 5 minutes. alert fired 15 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-wdpfr", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-wdpfr has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000001-hj882", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000001-hj882 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-5cffdb78dc-tz654", region="canadacentral", severity="warning" Description: Pod kube-system/hubble-relay-5cffdb78dc-tz654 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000002-5w7z2", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000002-5w7z2 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-cxdpf", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-cxdpf has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-vxj68", region="canadacentral", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-vxj68 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-11-13T20:41:05Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-41306043-vmss000000-84std", region="canadacentral", severity="warning" Description: Pod kube-system/eraser-aks-system-41306043-vmss000000-84std has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-t6cxk", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-t6cxk has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-sqzqm", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-sqzqm has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-11-13T20:43:06Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-gb9rv", region="canadacentral", severity="warning" Description: Pod kube-system/ama-metrics-node-gb9rv has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-78789f646f-2wdlw", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-78789f646f-2wdlw has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-78789f646f-z5ntg", region="canadacentral", severity="warning" Description: Pod sessiongate/sessiongate-78789f646f-z5ntg has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-wr8wf", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-wr8wf has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-11-13T20:44:07Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-zncb7", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-zncb7 has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-11-13T20:44:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3124352-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-69dffb97bc-9vgxw", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-69dffb97bc-9vgxw has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerRetryHotLoop firedfull failure pattern: alert [svc] BackendControllerRetryHotLoop fired Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | alert | 5 | 2.60%5 of 192 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | Show trend detailsNov 10: 2 · Nov 11: 2 · Nov 12: 1 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 6 time(s) Firing 1: State: Resolved Started: 2026-11-16T17:27:24Z Ended: 2026-11-16T17:59:21Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="placement", region="westus3", severity="warning" Description: Backend controller workqueue placement has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 2: State: Resolved Started: 2026-11-16T17:30:23Z Ended: 2026-11-16T17:58:24Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="identityroleassignments", region="westus3", severity="warning" Description: Backend controller workqueue identityroleassignments has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 3: State: Resolved Started: 2026-11-16T18:06:26Z Ended: 2026-11-16T18:19:21Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="ensuremanagedresourcegroup", region="westus3", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 4: State: Resolved Started: 2026-11-16T18:23:23Z Ended: 2026-11-16T18:52:22Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="placement", region="westus3", severity="warning" Description: Backend controller workqueue placement has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 5: State: Resolved Started: 2026-11-16T18:37:23Z Ended: 2026-11-16T18:49:25Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="identityroleassignments", region="westus3", severity="warning" Description: Backend controller workqueue identityroleassignments has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 6: State: Fired (not resolved) Started: 2026-11-16T18:57:25Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="backend", name="ensuremanagedresourcegroup", region="westus3", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-14T13:35:08Z Ended: 2026-11-14T13:49:00Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j7717504-svc", component="backend", name="ensuremanagedresourcegroup", region="centralus", severity="warning" Description: Backend controller workqueue ensuremanagedresourcegroup has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 4 time(s) Firing 1: State: Resolved Started: 2026-11-12T15:43:07Z Ended: 2026-11-12T15:54:07Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 2: State: Resolved Started: 2026-11-12T16:08:07Z Ended: 2026-11-12T16:16:07Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 3: State: Resolved Started: 2026-11-12T16:37:06Z Ended: 2026-11-12T16:50:06Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Firing 4: State: Fired (not resolved) Started: 2026-11-12T17:37:08Z Severity: Sev3 Labels: alertname="BackendControllerRetryHotLoop", cluster="ci01-j5531776-svc", component="backend", name="deleteorphanedcosmosresources", region="canadacentral", severity="warning" Description: Backend controller workqueue deleteorphanedcosmosresources has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Contributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: 429; provider Microsoft.ManagedIdentityfull failure pattern: ERROR CODE: 429; provider Microsoft.ManagedIdentity Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | e2e | 5 | 2.60%5 of 192 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 1 · Nov 14: 1 · Nov 15: 0 · Nov 16: 2 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x303d4ee7620>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: c7391ef9-779f-4fec-a1fc-92e1977eada5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x303d4ee7620>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-03-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: c7391ef9-779f-4fec-a1fc-92e1977eada5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0xc427f5a17d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: bbc0aa39-601e-43c7-8d35-95370cb55622
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:101]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0xc427f5a17d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-47/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: bbc0aa39-601e-43c7-8d35-95370cb55622
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:112]: failed to get cluster-api-azure managed identity
Unexpected error:
<*exported.ResponseError | 0x1ad1a5ffb110>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
correlation ID: f6f33d77-6df0-446b-9f77-5c085c8f1737
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:112]: failed to get cluster-api-azure managed identity
Unexpected error:
<*exported.ResponseError | 0x1ad1a5ffb110>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard0-02-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
correlation ID: f6f33d77-6df0-446b-9f77-5c085c8f1737
client request ID: unavailable
occurredContributing tests (1)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-<id>": nodes "cluster-np-<id>" is forbidden: autoscaling.openshift.io/ManagedNode Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 5 | 2.60%5 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 4 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-5-0-fvh62c"
Unexpected error:
<*errors.joinError | 0xe9c5185e558>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-fvh62c-qp98f-4xr6k": nodes "cluster-np-fvh62c-qp98f-4xr6k" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xe9c50b2e640>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e620>{
msg: "failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e600>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xe9c518f6f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-fvh62c-qp98f-4xr6k",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-5-0-fvh62c"
Unexpected error:
<*errors.joinError | 0xe9c5185e558>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-fvh62c-qp98f-4xr6k": nodes "cluster-np-fvh62c-qp98f-4xr6k" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0xe9c50b2e640>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e620>{
msg: "failed to label node cluster-np-fvh62c-qp98f-4xr6k egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0xe9c50b2e600>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-fvh62c-qp98f-4xr6k\": nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0xe9c518f6f00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-fvh62c-qp98f-4xr6k\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-fvh62c-qp98f-4xr6k",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-xcrqj2"
Unexpected error:
<*errors.joinError | 0x1ff6e5be7bd8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xcrqj2-s4s2f-4fg9q": nodes "cluster-np-xcrqj2-s4s2f-4fg9q" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1ff6e6292100>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920e0>{
msg: "failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920c0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1ff6e62961e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xcrqj2-s4s2f-4fg9q",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-xcrqj2"
Unexpected error:
<*errors.joinError | 0x1ff6e5be7bd8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-xcrqj2-s4s2f-4fg9q": nodes "cluster-np-xcrqj2-s4s2f-4fg9q" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x1ff6e6292100>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920e0>{
msg: "failed to label node cluster-np-xcrqj2-s4s2f-4fg9q egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x1ff6e62920c0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-xcrqj2-s4s2f-4fg9q\": nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x1ff6e62961e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-xcrqj2-s4s2f-4fg9q\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-xcrqj2-s4s2f-4fg9q",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7z7qh5"
Unexpected error:
<*errors.joinError | 0x3ff8850ea50>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-7z7qh5-5kzdr-5p27s": nodes "cluster-np-7z7qh5-5kzdr-5p27s" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x3ff87627d80>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627d00>{
msg: "failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627cc0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x3ff884aeb40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-7z7qh5-5kzdr-5p27s",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7z7qh5"
Unexpected error:
<*errors.joinError | 0x3ff8850ea50>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-7z7qh5-5kzdr-5p27s": nodes "cluster-np-7z7qh5-5kzdr-5p27s" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x3ff87627d80>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627d00>{
msg: "failed to label node cluster-np-7z7qh5-5kzdr-5p27s egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x3ff87627cc0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-7z7qh5-5kzdr-5p27s\": nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x3ff884aeb40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-7z7qh5-5kzdr-5p27s\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-7z7qh5-5kzdr-5p27s",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (5) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.56%3 of 192 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsNov 10: 0 · Nov 11: 54 · Nov 12: 21 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:153]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x10c27adac5d0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-whdb88zlcngg" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-whdb88zlcngg, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-whdb88zlcngg\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-whdb88zlcngg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x10c27a5d1c30>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-whdb88zlcngg",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:153]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x10c27adac5d0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-whdb88zlcngg" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-whdb88zlcngg, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-whdb88zlcngg\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-whdb88zlcngg, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x10c27a5d1c30>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-whdb88zlcngg",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapErrors | 0x1923f3bc0b70>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jn9zxd8bpzrt" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-jn9zxd8bpzrt\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1923f3d1cb00>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapErrors | 0x1923f3bc0b70>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-jn9zxd8bpzrt" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cilium-cluster\" in resourcegroup=\"complex-cilium-kv-jn9zxd8bpzrt\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1923f3d1cb00>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-jn9zxd8bpzrt",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapErrors | 0x3a9a18474690>:
failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-jkq8brgglgg7" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded
{
msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7",
},
<*fmt.wrapErrors | 0x3a9a18474660>{
msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>0x3a9a185ffe50 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapErrors | 0x3a9a18474690>:
failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-jkq8brgglgg7" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded
{
msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7",
},
<*fmt.wrapErrors | 0x3a9a18474660>{
msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-jkq8brgglgg7\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster engineering-hcp-cluster in resource group kusto-logs-jkq8brgglgg7, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a9a185ffe50>0x3a9a185ffe50 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (59)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 3 | 1.56%3 of 192 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x4f2057fa1c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
err: <*verifiers.pollObservationError | 0x4f2057fa180>{
observation: <*errors.errorString | 0x4f2037c8bd0>{
s: "dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
},
interruption: <*fmt.wrapError | 0x4f2057fa160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x4f203456140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x4f2057fa140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x4f205896180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x4f2057fa1c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6hgjjvzl5r57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
err: <*verifiers.pollObservationError | 0x4f2057fa180>{
observation: <*errors.errorString | 0x4f2037c8bd0>{
s: "dev/westus3/ci01-j1435008-mgmt-1/ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster router=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001: expected matching summary after test start and within 8m; summaryUID=13be42f5-6671-4a26-9e64-92c7ab7cd823 node=aks-userswft2-31957569-vmss000001 pass=f96b57a2-519d-41d2-a75a-99c7c360275e at=2026-11-16T10:19:25Z",
},
interruption: <*fmt.wrapError | 0x4f2057fa160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x4f203456140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x4f2057fa140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j1435008-mgmt-1\"; let ns = \"ocm-arohcpci01-2u48ej8b17o3gq7vc2tfcghlj5e8nscf-cluster\"; let start = datetime(2026-11-16T10:03:45.9661662Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x4f205896180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2387254f8000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x238726e308c0>{
s: "dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2387254f8000>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jj2hc55dbhml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*errors.errorString | 0x238726e308c0>{
s: "dev/canadacentral/ci01-j6661376-mgmt-2/ocm-arohcpci01-2u2b0n3fmhnbnca9o2nl1t6qca991sir-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1eea1ef05600>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
err: <*verifiers.pollObservationError | 0x1eea1ef055e0>{
observation: <*errors.errorString | 0x1eea20b3b080>{
s: "dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
},
interruption: <*fmt.wrapError | 0x1eea1ef055c0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1eea1ee206e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1eea1ef05560>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1eea2040eea0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1eea1ef05600>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-lwch49mnklv7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
err: <*verifiers.pollObservationError | 0x1eea1ef055e0>{
observation: <*errors.errorString | 0x1eea20b3b080>{
s: "dev/westus3/ci01-j6558976-mgmt-2/ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster router=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000: expected matching summary after test start and within 8m; summaryUID=42d4e768-000f-46c2-aa60-645d36f48ce7 node=aks-userswft2-19060083-vmss000000 pass=efabbc47-17ec-44ac-999d-3d2c9503d640 at=2026-11-12T03:59:40Z",
},
interruption: <*fmt.wrapError | 0x1eea1ef055c0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1eea1ee206e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1eea1ef05560>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6558976-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1eh521c6f4vt1o80ab8sa9jst86fqk-cluster\"; let start = datetime(2026-11-12T03:45:13.7484373Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1eea2040eea0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 3 | 1.56%3 of 192 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Resolved Started: 2026-11-16T18:09:10Z Ended: 2026-11-16T18:26:09Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6261632-svc", component="backend", name="clusterdeletioncontroller", region="westus3", severity="warning" Description: Backend controller workqueue clusterdeletioncontroller has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-11-16T18:09:10Z Ended: 2026-11-16T18:26:09Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6261632-svc", component="backend", name="clusterchildresourcescleanupcontroller", region="westus3", severity="warning" Description: Backend controller workqueue clusterchildresourcescleanupcontroller has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 3: State: Fired (not resolved) Started: 2026-11-16T19:12:07Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6261632-svc", component="backend", name="clusterchildresourcescleanupcontroller", region="westus3", severity="warning" Description: Backend controller workqueue clusterchildresourcescleanupcontroller has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 4: State: Fired (not resolved) Started: 2026-11-16T19:14:10Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6261632-svc", component="backend", name="clusterdeletioncontroller", region="westus3", severity="warning" Description: Backend controller workqueue clusterdeletioncontroller has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-11-12T23:02:19Z Ended: 2026-11-12T23:11:19Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2715136-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-11-12T23:02:19Z Ended: 2026-11-12T23:11:19Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2715136-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-11T02:18:36Z Ended: 2026-11-11T02:34:34Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4684288-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
scheduling requested CPU must be positivefull failure pattern: scheduling requested CPU must be positive Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 3 | 1.56%3 of 192 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsNov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.000s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 1 scheduling requested CPU must be positive
Expected
<bool>: true
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.000s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 1 scheduling requested CPU must be positive
Expected
<bool>: true
to be falsefail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
<bool>: true
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 2 scheduling requested CPU must be positive
Expected
<bool>: true
to be falsefail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 1 scheduling requested CPU must be positive
Expected
<bool>: true
to be false
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s.
scheduling data was not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with:
stamp 1 scheduling requested CPU must be positive
Expected
<bool>: true
to be falseContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGridfull failure pattern: ERROR CODE: DeploymentFailed; detail code Timeout; detail message Operation is being Canceled due to timeout.; provider Microsoft.EventGrid Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | provision | 3 | 1.56%3 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 10: 3 · Nov 11: 3 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-11-11T14:26:38.768Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T14:26:40.146Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:26:42.295Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:26:46.642Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:26:55.498Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:27:12.634Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:27:45.036Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b?api-version=2021-04-01"
time=2026-11-11T14:27:45.036Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T14:27:46.265Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9432960%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fa5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b
time=2026-11-11T14:43:19.581Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/a5168dff8e3058d12d75a1e386a15d1381c5b0ccff5e5f2a22845ac4a159f35b/operationStatuses/08584071996199689084\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 64f93b66-743f-4a72-adcd-148ae5d247c0\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9432960/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 64f93b66-743f-4a72-adcd-148ae5d247c0\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9432960/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j9432960\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-11T00:47:48.881Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T00:47:50.316Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:47:52.553Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:47:56.856Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:05.395Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:22.613Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:57.747Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908?api-version=2021-04-01"
time=2026-11-11T00:48:57.748Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T00:48:58.504Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3478912%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908
time=2026-11-11T01:04:33.988Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/16f64297f896548673add9560433f6c5a588752c5eae8820cc20765325fb6908/operationStatuses/08584072487474526673\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2e23547e-99e6-4dbe-b935-a08d562df87a\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3478912/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 2e23547e-99e6-4dbe-b935-a08d562df87a\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3478912/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j3478912\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-11T00:47:18.513Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T00:47:20.111Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:22.298Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:26.447Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:34.929Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:47:51.119Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:48:23.772Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81?api-version=2021-04-01"
time=2026-11-11T00:48:23.772Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T00:48:24.640Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2734720%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fb1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81
time=2026-11-11T01:03:58.863Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/b1aaf16a6e9ab0413bd8422378cc92ea974a1a5e7ebb53dd866ba0a4eceeef81/operationStatuses/08584072487812695254\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: bc3822da-78e4-4890-b47c-d63a893c752f\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2734720/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: bc3822da-78e4-4890-b47c-d63a893c752f\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2734720/providers/Microsoft.EventGrid/namespaces/arohcp-ci01-maestro-j2734720\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Timeout\\\",\\r\\n \\\"message\\\": \\\"Operation is being Canceled due to timeout.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] FleetControllerRetryHotLoop firedfull failure pattern: alert [svc] FleetControllerRetryHotLoop fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | alert | 3 | 1.56%3 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 2 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-16T17:22:40Z Severity: Sev3 Labels: alertname="FleetControllerRetryHotLoop", cluster="ci01-j6261632-svc", component="fleet", name="clustersserviceregistrationcontroller", region="westus3", severity="warning" Description: Fleet controller workqueue clustersserviceregistrationcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-11T21:49:58Z Severity: Sev3 Labels: alertname="FleetControllerRetryHotLoop", cluster="ci01-j3946112-svc", component="fleet", name="clustersserviceregistrationcontroller", region="canadacentral", severity="warning" Description: Fleet controller workqueue clustersserviceregistrationcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-11T17:49:52Z Ended: 2026-11-11T18:15:55Z Severity: Sev3 Labels: alertname="FleetControllerRetryHotLoop", cluster="ci01-j4027392-svc", component="fleet", name="nodepoolcontroller", region="westus3", severity="warning" Description: Fleet controller workqueue nodepoolcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work. Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operati...full failure pattern: ERROR CODE: DeploymentFailed; detail code RequestTimeout; detail message Database account creation failed. Operation Id: <uuid>, Error : Message: {"Errors":["The requested operation exceeded maximum alloted time. Learn more: <url>; provider Microsoft.DocumentDBSignal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | provision | 2 | 1.04%2 of 192 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)time=2026-11-12T02:15:15.432Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-12T02:15:18.214Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:20.478Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:24.642Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:33.477Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:15:50.325Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:16:22.712Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6?api-version=2021-04-01"
time=2026-11-12T02:16:22.712Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-12T02:16:23.692Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4812544%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6
time=2026-11-12T02:22:55.890Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/f6c014d72bb93a7cc491d914dc2b3ead16146c433f8234c766aaa84f0bb4aaa6/operationStatuses/08584071571022698221\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8e807b60-5023-4afe-b65a-da1bd577fad4\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4812544/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 8e807b60-5023-4afe-b65a-da1bd577fad4\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4812544/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j4812544\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: 49b4ee9d-91eb-463e-be2a-f9a6a2f0c36a, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: b3cae6cf-1e2a-46fe-90b3-6a504a79fd6f, Request URI: /apps/457b0b96-1859-441b-9897-a8af2244ff42/services/0882f1b4-9e5e-44a9-a23d-5aff02eaa165/partitions/d4426837-d7ca-44b7-a111-280431846736/replicas/134337933848286181p, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-12T02:22:03.6687109Z, RequestEndTime: 2026-11-12T02:22:09.6687149Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:15.3229628Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.585,\\\\\\\"memory\\\\\\\":23308184.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1208,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":119},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:25.3386251Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.854,\\\\\\\"memory\\\\\\\":23300812.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0732,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:35.3543979Z\\\\\\\",\\\\\\\"cpu\\\\\\\":0.975,\\\\\\\"memory\\\\\\\":23304432.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0617,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:45.3560901Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.387,\\\\\\\"memory\\\\\\\":23307736.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1176,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:21:55.3718002Z\\\\\\\",\\\\\\\"cpu\\\\\\\":1.442,\\\\\\\"memory\\\\\\\":23304548.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1254,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-12T02:22:05.3874211Z\\\\\\\",\\\\\\\"cpu\\\\\\\":7.766,\\\\\\\"memory\\\\\\\":23297692.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1201,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":132}]}\\\\r\\\\nRequestStart: 2026-11-12T02:22:03.6687109Z; ResponseTime: 2026-11-12T02:22:09.6687149Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be187.documents.azure.com:11005/apps/457b0b96-1859-441b-9897-a8af2244ff42/services/0882f1b4-9e5e-44a9-a23d-5aff02eaa165/partitions/d4426837-d7ca-44b7-a111-280431846736/replicas/134337933848286181p, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5994.097, ActivityId: b3cae6cf-1e2a-46fe-90b3-6a504a79fd6f, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11005 | status: Connected | lkt: 11/12/2026 2:22:09 AM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6687109Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0052},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6687161Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0176},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6687337Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.3427},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:03.6690764Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5995.1448},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:09.6642212Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1011},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-12T02:22:09.6643223Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-12T02:19:50.3213621Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-12T02:19:50.3213621Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-12T02:19:55.6660542Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-11-11T20:21:34.625Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam"
time=2026-11-11T20:21:36.427Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:21:38.522Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:21:42.812Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:21:51.669Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:22:09.004Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:22:43.129Z level=INFO msg="transient 404 DeploymentNotFound on deployment GET" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra url="https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f?api-version=2021-04-01"
time=2026-11-11T20:22:43.130Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-11-11T20:22:44.191Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0985344%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f
time=2026-11-11T20:30:46.072Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/bcbe4e59d84c11b02a1fb3476a9f40fc2f9de673767fcf6e005929d1a312ad3f/operationStatuses/08584071783219952338\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b161dda7-fd92-46b2-b2e8-4ab8c9119d2c\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0985344/providers/Microsoft.Resources/deployments/rp-cosmos-account\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: b161dda7-fd92-46b2-b2e8-4ab8c9119d2c\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0985344/providers/Microsoft.DocumentDB/databaseAccounts/arohcpci01-rp-j0985344\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RequestTimeout\\\",\\r\\n \\\"message\\\": \\\"Database account creation failed. Operation Id: d1ede56b-38f5-4425-b382-a754f12b45fa, Error : Message: {\\\\\\\"Errors\\\\\\\":[\\\\\\\"The requested operation exceeded maximum alloted time. Learn more: https://aka.ms/cosmosdb-tsg-service-request-timeout\\\\\\\"]}\\\\r\\\\nActivityId: db332a6f-d6ca-4b3d-9da4-e65e2b4782a6, Request URI: /apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/326dacfa-a565-4b95-ab41-35500c782898/partitions/1a776f41-28f7-4e9f-8cb1-4d48f482ec6d/replicas/134384651191442271s, RequestStats: \\\\r\\\\nRequestStartTime: 2026-11-11T20:29:52.3082641Z, RequestEndTime: 2026-11-11T20:29:57.3086766Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\"systemHistory\\\\\\\":[{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:06.3166707Z\\\\\\\",\\\\\\\"cpu\\\\\\\":7.488,\\\\\\\"memory\\\\\\\":23682836.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.1221,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:16.3324546Z\\\\\\\",\\\\\\\"cpu\\\\\\\":4.524,\\\\\\\"memory\\\\\\\":23665912.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0709,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:26.3382658Z\\\\\\\",\\\\\\\"cpu\\\\\\\":4.219,\\\\\\\"memory\\\\\\\":23659192.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0824,\\\\\\\"availableThreads\\\\\\\":32762,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:36.3540698Z\\\\\\\",\\\\\\\"cpu\\\\\\\":7.313,\\\\\\\"memory\\\\\\\":23651792.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0979,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:46.3550310Z\\\\\\\",\\\\\\\"cpu\\\\\\\":16.367,\\\\\\\"memory\\\\\\\":23598600.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.0665,\\\\\\\"availableThreads\\\\\\\":32764,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122},{\\\\\\\"dateUtc\\\\\\\":\\\\\\\"2026-11-11T20:29:56.3555696Z\\\\\\\",\\\\\\\"cpu\\\\\\\":5.060,\\\\\\\"memory\\\\\\\":23606476.000,\\\\\\\"threadInfo\\\\\\\":{\\\\\\\"isThreadStarving\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"threadWaitIntervalInMs\\\\\\\":0.096,\\\\\\\"availableThreads\\\\\\\":32765,\\\\\\\"minThreads\\\\\\\":8,\\\\\\\"maxThreads\\\\\\\":32767},\\\\\\\"numberOfOpenTcpConnection\\\\\\\":122}]}\\\\r\\\\nRequestStart: 2026-11-11T20:29:52.3082641Z; ResponseTime: 2026-11-11T20:29:57.3086766Z; StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-canadacentral1-be149.documents.azure.com:11312/apps/b24b854e-3ba8-4cff-9e2f-ec9105cfcd25/services/326dacfa-a565-4b95-ab41-35500c782898/partitions/1a776f41-28f7-4e9f-8cb1-4d48f482ec6d/replicas/134384651191442271s, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: null, BELatencyMs: 5002.075, ActivityId: db332a6f-d6ca-4b3d-9da4-e65e2b4782a6, RetryAfterInMs: , globalNRegionCommittedGLSN: -1, ReplicaHealthStatuses: [(port: 11312 | status: Connected | lkt: 11/11/2026 8:29:57 PM)], TransportRequestTimeline: {\\\\\\\"requestTimeline\\\\\\\":[{\\\\\\\"event\\\\\\\": \\\\\\\"Created\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3082641Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.0037},{\\\\\\\"event\\\\\\\": \\\\\\\"ChannelAcquisitionStarted\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3082678Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.02},{\\\\\\\"event\\\\\\\": \\\\\\\"Pipelined\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3082878Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.2288},{\\\\\\\"event\\\\\\\": \\\\\\\"Transit Time\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:52.3085166Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 5002.673},{\\\\\\\"event\\\\\\\": \\\\\\\"Received\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:57.3111896Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0.1463},{\\\\\\\"event\\\\\\\": \\\\\\\"Completed\\\\\\\", \\\\\\\"startTimeUtc\\\\\\\": \\\\\\\"2026-11-11T20:29:57.3113359Z\\\\\\\", \\\\\\\"durationInMs\\\\\\\": 0}],\\\\\\\"serviceEndpointStats\\\\\\\":{\\\\\\\"inflightRequests\\\\\\\":1,\\\\\\\"openConnections\\\\\\\":1},\\\\\\\"connectionStats\\\\\\\":{\\\\\\\"waitforConnectionInit\\\\\\\":\\\\\\\"False\\\\\\\",\\\\\\\"callsPendingReceive\\\\\\\":0,\\\\\\\"lastSendAttempt\\\\\\\":\\\\\\\"2026-11-11T20:27:38.9692278Z\\\\\\\",\\\\\\\"lastSend\\\\\\\":\\\\\\\"2026-11-11T20:27:38.9692278Z\\\\\\\",\\\\\\\"lastReceive\\\\\\\":\\\\\\\"2026-11-11T20:27:39.0004715Z\\\\\\\"},\\\\\\\"requestSizeInBytes\\\\\\\":875,\\\\\\\"responseMetadataSizeInBytes\\\\\\\":30,\\\\\\\"responseBodySizeInBytes\\\\\\\":133};\\\\r\\\\n ResourceType: Replica, OperationType: ForceConfigRefresh\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [svc] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | alert | 2 | 1.04%2 of 192 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-12T16:02:47Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-jpbc4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-12T16:02:48Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-jpbc4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-11T19:13:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-hbqnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-11-11T19:13:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="arohcp-monitor-kube-state-metrics-6b5dd9555f-hbqnj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment aro-hcp/aro-hcp-backend has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | alert | 2 | 1.04%2 of 192 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsNov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)alert fired 6 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-12T15:52:46Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-j4rtb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f51ff0e8-0932-44b5-86a1-12ea492c373f" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-j4rtb (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-11-12T15:52:46Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-j4rtb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f51ff0e8-0932-44b5-86a1-12ea492c373f" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-j4rtb (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-11-12T16:01:46Z Ended: 2026-11-12T16:57:57Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Resolved Started: 2026-11-12T16:01:46Z Ended: 2026-11-12T16:57:48Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 5: State: Fired (not resolved) Started: 2026-11-12T17:13:47Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 6: State: Fired (not resolved) Started: 2026-11-12T17:13:47Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5531776-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.43:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5531776/providers/microsoft.monitor/accounts/services-j5531776", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-6c9dd84bc7-8ldkw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a6f2ab15-a7ad-465a-9e85-1518697507ec" Description: Pod aro-hcp/aro-hcp-backend-6c9dd84bc7-8ldkw (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). alert fired 6 time(s) Firing 1: State: Fired (not resolved) Started: 2026-11-11T19:17:09Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-88b69", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="468e8019-4d61-44d8-a033-41db4b1a7b1b" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-88b69 (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-11-11T19:17:09Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-88b69", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="468e8019-4d61-44d8-a033-41db4b1a7b1b" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-88b69 (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-11-11T19:29:14Z Ended: 2026-11-11T19:35:10Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Resolved Started: 2026-11-11T19:29:14Z Ended: 2026-11-11T19:35:10Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 5: State: Fired (not resolved) Started: 2026-11-11T20:34:08Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Firing 6: State: Fired (not resolved) Started: 2026-11-11T20:34:08Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0146816-svc", component="kubernetes-infrastructure", container="aro-hcp-backend", endpoint="http", environment="ci01", instance="10.128.64.71:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0146816/providers/microsoft.monitor/accounts/services-j0146816", microsoft.metrics_include_label="service", namespace="aro-hcp", pod="aro-hcp-backend-b8b684845-dnxxg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a816491e-e9a1-4eb6-be75-876cd45bd797" Description: Pod aro-hcp/aro-hcp-backend-b8b684845-dnxxg (aro-hcp-backend) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: AROHCPCapacityHeavyUse; detail message ARO HCP is currently experiencing capacity constraints. Try again later.; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: AROHCPCapacityHeavyUse; detail message ARO HCP is currently experiencing capacity constraints. Try again later.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 70 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-28z5zb with version 4.20.47
Unexpected error:
<*fmt.wrapError | 0xb16b9eba1e0>:
failed to create HCP cluster np-version-upgrade-cluster-28z5zb: failed waiting for cluster="np-version-upgrade-cluster-28z5zb" in resourcegroup="rg-np-version-upgrade-28z5zb-tbh7t6q7rs6v" to finish creating: GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed073fc1-3405-45f4-8445-d5087793a6ca
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: AROHCPCapacityHeavyUse
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed073fc1-3405-45f4-8445-d5087793a6ca",
"name": "ed073fc1-3405-45f4-8445-d5087793a6ca",
"status": "Failed",
"startTime": "2026-11-16T17:17:42.231729415Z",
"endTime": "2026-11-16T17:46:44.360987247Z",
"error": {
"code": "AROHCPCapacityHeavyUse",
"message": "ARO HCP is currently experiencing capacity constraints. Try again later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed073fc1-3405-45f4-8445-d5087793a6ca
correlation ID: 169bf67f-1506-4324-a038-d3dc55fdd5af
client request ID: b72bd62b-911b-45c3-9c05-b636714b2286
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-28z5zb with version 4.20.47
Unexpected error:
<*fmt.wrapError | 0xb16b9eba1e0>:
failed to create HCP cluster np-version-upgrade-cluster-28z5zb: failed waiting for cluster="np-version-upgrade-cluster-28z5zb" in resourcegroup="rg-np-version-upgrade-28z5zb-tbh7t6q7rs6v" to finish creating: GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed073fc1-3405-45f4-8445-d5087793a6ca
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: AROHCPCapacityHeavyUse
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed073fc1-3405-45f4-8445-d5087793a6ca",
"name": "ed073fc1-3405-45f4-8445-d5087793a6ca",
"status": "Failed",
"startTime": "2026-11-16T17:17:42.231729415Z",
"endTime": "2026-11-16T17:46:44.360987247Z",
"error": {
"code": "AROHCPCapacityHeavyUse",
"message": "ARO HCP is currently experiencing capacity constraints. Try again later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ed073fc1-3405-45f4-8445-d5087793a6ca
correlation ID: 169bf67f-1506-4324-a038-d3dc55fdd5af
client request ID: b72bd62b-911b-45c3-9c05-b636714b2286
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:102]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x663480242a0>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-pp5lx768bk22" to finish creating: GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6331910a-b625-4b85-aafc-6ae7b4220580
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: AROHCPCapacityHeavyUse
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6331910a-b625-4b85-aafc-6ae7b4220580",
"name": "6331910a-b625-4b85-aafc-6ae7b4220580",
"status": "Failed",
"startTime": "2026-11-16T17:17:21.258353887Z",
"endTime": "2026-11-16T17:46:24.36686246Z",
"error": {
"code": "AROHCPCapacityHeavyUse",
"message": "ARO HCP is currently experiencing capacity constraints. Try again later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6331910a-b625-4b85-aafc-6ae7b4220580
correlation ID: e4d7b1dd-7a47-41ba-b149-eb8440386f05
client request ID: 991fbd13-775f-4d41-ae76-f62aca6281ec
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:102]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x663480242a0>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-pp5lx768bk22" to finish creating: GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6331910a-b625-4b85-aafc-6ae7b4220580
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: AROHCPCapacityHeavyUse
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6331910a-b625-4b85-aafc-6ae7b4220580",
"name": "6331910a-b625-4b85-aafc-6ae7b4220580",
"status": "Failed",
"startTime": "2026-11-16T17:17:21.258353887Z",
"endTime": "2026-11-16T17:46:24.36686246Z",
"error": {
"code": "AROHCPCapacityHeavyUse",
"message": "ARO HCP is currently experiencing capacity constraints. Try again later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6331910a-b625-4b85-aafc-6ae7b4220580
correlation ID: e4d7b1dd-7a47-41ba-b149-eb8440386f05
client request ID: 991fbd13-775f-4d41-ae76-f62aca6281ec
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:114]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x2319e04986a0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nfmvzp4t8xxs" to finish creating: GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: AROHCPCapacityHeavyUse
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0",
"name": "8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0",
"status": "Failed",
"startTime": "2026-11-16T18:12:56.466549268Z",
"endTime": "2026-11-16T18:41:58.122900442Z",
"error": {
"code": "AROHCPCapacityHeavyUse",
"message": "ARO HCP is currently experiencing capacity constraints. Try again later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0
correlation ID: f1e6d3e4-ec4b-43df-9af7-57e31cf5d936
client request ID: 860159c9-497f-4611-ac75-e2f430712109
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:114]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x2319e04986a0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-nfmvzp4t8xxs" to finish creating: GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: AROHCPCapacityHeavyUse
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0",
"name": "8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0",
"status": "Failed",
"startTime": "2026-11-16T18:12:56.466549268Z",
"endTime": "2026-11-16T18:41:58.122900442Z",
"error": {
"code": "AROHCPCapacityHeavyUse",
"message": "ARO HCP is currently experiencing capacity constraints. Try again later."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6261632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b3b7d9b-4cd9-43b0-acfb-1d1ed935c7a0
correlation ID: f1e6d3e4-ec4b-43df-9af7-57e31cf5d936
client request ID: 860159c9-497f-4611-ac75-e2f430712109
occurredContributing tests (71)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster placement did not complete before the deadline; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message cluster placement did not complete before the deadline; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsNov 10: 5 · Nov 11: 16 · Nov 12: 31 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:124]: failed to create fully private HCP cluster "full-priv-kv" with private KV
Unexpected error:
<*fmt.wrapError | 0x2de5b2594880>:
failed waiting for cluster="full-priv-kv" in resourcegroup="full-priv-kv-4ccjmts62txv" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"name": "a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"status": "Failed",
"startTime": "2026-11-12T15:34:34.191986492Z",
"endTime": "2026-11-12T16:04:22.917130532Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
correlation ID: 11162cd7-dc7c-4515-9a6f-6e443f7b1a95
client request ID: 89f08d01-5d98-4385-a502-d2c0f1b98c7f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:124]: failed to create fully private HCP cluster "full-priv-kv" with private KV
Unexpected error:
<*fmt.wrapError | 0x2de5b2594880>:
failed waiting for cluster="full-priv-kv" in resourcegroup="full-priv-kv-4ccjmts62txv" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"name": "a4eb80cd-da90-4650-ae52-e099f5dcdbfa",
"status": "Failed",
"startTime": "2026-11-12T15:34:34.191986492Z",
"endTime": "2026-11-12T16:04:22.917130532Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a4eb80cd-da90-4650-ae52-e099f5dcdbfa
correlation ID: 11162cd7-dc7c-4515-9a6f-6e443f7b1a95
client request ID: 89f08d01-5d98-4385-a502-d2c0f1b98c7f
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-h6k5jr" with version 4.22.21 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3d89aea50b00>:
failed to create HCP cluster cluster-zstream-4-22-h6k5jr: failed waiting for cluster="cluster-zstream-4-22-h6k5jr" in resourcegroup="rg-zstream-upgrade-4-22-vdsknjqsrbfz" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"name": "7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"status": "Failed",
"startTime": "2026-11-12T15:33:13.01421483Z",
"endTime": "2026-11-12T16:07:07.952160345Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
correlation ID: 314f2449-0ae2-47ef-b9fd-89f78927292e
client request ID: 92baffe5-5311-4a79-a62c-25eb2f9fb38c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-h6k5jr" with version 4.22.21 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3d89aea50b00>:
failed to create HCP cluster cluster-zstream-4-22-h6k5jr: failed waiting for cluster="cluster-zstream-4-22-h6k5jr" in resourcegroup="rg-zstream-upgrade-4-22-vdsknjqsrbfz" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"name": "7502bdb0-6f3f-4e31-ba9c-8333a138d9db",
"status": "Failed",
"startTime": "2026-11-12T15:33:13.01421483Z",
"endTime": "2026-11-12T16:07:07.952160345Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7502bdb0-6f3f-4e31-ba9c-8333a138d9db
correlation ID: 314f2449-0ae2-47ef-b9fd-89f78927292e
client request ID: 92baffe5-5311-4a79-a62c-25eb2f9fb38c
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tfw224" with version 4.23.0-ec.1 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x377271de84a0>:
failed to create HCP cluster cluster-zstream-4-23-tfw224: failed waiting for cluster="cluster-zstream-4-23-tfw224" in resourcegroup="rg-zstream-upgrade-4-23-bn224qrf75xd" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec",
"name": "22c29099-f38e-42a3-9a4c-90084ad155ec",
"status": "Failed",
"startTime": "2026-11-12T15:32:40.053212969Z",
"endTime": "2026-11-12T16:01:50.625591398Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
correlation ID: 40d57c20-8bbf-48b4-8792-08c7b3124280
client request ID: 48105e19-c842-4e4b-bcc5-da59cee3b1bf
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tfw224" with version 4.23.0-ec.1 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x377271de84a0>:
failed to create HCP cluster cluster-zstream-4-23-tfw224: failed waiting for cluster="cluster-zstream-4-23-tfw224" in resourcegroup="rg-zstream-upgrade-4-23-bn224qrf75xd" to finish creating: GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec",
"name": "22c29099-f38e-42a3-9a4c-90084ad155ec",
"status": "Failed",
"startTime": "2026-11-12T15:32:40.053212969Z",
"endTime": "2026-11-12T16:01:50.625591398Z",
"error": {
"code": "InternalServerError",
"message": "cluster placement did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5531776.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/22c29099-f38e-42a3-9a4c-90084ad155ec
correlation ID: 40d57c20-8bbf-48b4-8792-08c7b3124280
client request ID: 48105e19-c842-4e4b-bcc5-da59cee3b1bf
occurredContributing tests (39)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 2 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:210]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x22c18c6b24a0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-5hrn9mspwfjd" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12",
"name": "83b4c400-775d-4ffd-9b67-6287332ebc12",
"status": "Failed",
"startTime": "2026-11-14T13:07:18.872762914Z",
"endTime": "2026-11-14T13:36:22.180780509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
correlation ID: 8b44e4e2-2314-4db7-8d11-8d49e51bd3b0
client request ID: 07dcbf36-1afc-4c8f-8c6d-97deadcb3181
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:210]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x22c18c6b24a0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-5hrn9mspwfjd" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12",
"name": "83b4c400-775d-4ffd-9b67-6287332ebc12",
"status": "Failed",
"startTime": "2026-11-14T13:07:18.872762914Z",
"endTime": "2026-11-14T13:36:22.180780509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83b4c400-775d-4ffd-9b67-6287332ebc12
correlation ID: 8b44e4e2-2314-4db7-8d11-8d49e51bd3b0
client request ID: 07dcbf36-1afc-4c8f-8c6d-97deadcb3181
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:81]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x2c409df90980>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-q8h7nh5ldb6w" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc",
"name": "6302990a-b328-4ae4-97f7-38baacded1bc",
"status": "Failed",
"startTime": "2026-11-14T13:04:36.402299697Z",
"endTime": "2026-11-14T13:33:42.521828312Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
correlation ID: 1feffd8c-99db-457e-ada3-5a60ca87a066
client request ID: 124c0023-a5a5-4e28-9aad-bd6fd1f2656a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:81]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x2c409df90980>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-q8h7nh5ldb6w" to finish creating: GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc",
"name": "6302990a-b328-4ae4-97f7-38baacded1bc",
"status": "Failed",
"startTime": "2026-11-14T13:04:36.402299697Z",
"endTime": "2026-11-14T13:33:42.521828312Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7717504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6302990a-b328-4ae4-97f7-38baacded1bc
correlation ID: 1feffd8c-99db-457e-ada3-5a60ca87a066
client request ID: 124c0023-a5a5-4e28-9aad-bd6fd1f2656a
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:293]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x1f91d1e347c0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-qdvhlcv5hhb9" to finish creating: GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a",
"name": "b909bf18-c4a9-46c0-b023-9eed9656600a",
"status": "Failed",
"startTime": "2026-11-13T19:30:06.636680304Z",
"endTime": "2026-11-13T19:59:07.03404368Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
correlation ID: 7f002ae4-4687-4f1e-a00b-6530f9b7a9f9
client request ID: 9d864843-a38b-4d44-be3f-c3a6e910cb19
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:293]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x1f91d1e347c0>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-qdvhlcv5hhb9" to finish creating: GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a",
"name": "b909bf18-c4a9-46c0-b023-9eed9656600a",
"status": "Failed",
"startTime": "2026-11-13T19:30:06.636680304Z",
"endTime": "2026-11-13T19:59:07.03404368Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7356544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b909bf18-c4a9-46c0-b023-9eed9656600a
correlation ID: 7f002ae4-4687-4f1e-a00b-6530f9b7a9f9
client request ID: 9d864843-a38b-4d44-be3f-c3a6e910cb19
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 2 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2079e6d0e3e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
err: <*verifiers.pollObservationError | 0x2079e6d0e360>{
observation: <*errors.errorString | 0x2079e699e7c0>{
s: "dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
},
interruption: <*fmt.wrapError | 0x2079e6d0e340>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2079e6b8e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2079e6d0e300>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2079e80a8180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2079e6d0e3e0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zcfp2hgxbwzt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
err: <*verifiers.pollObservationError | 0x2079e6d0e360>{
observation: <*errors.errorString | 0x2079e699e7c0>{
s: "dev/centralus/ci01-j8436224-mgmt-1/ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster: expected all current routers >= desired replicas > 0; desired=0 current=3 rows=3",
},
interruption: <*fmt.wrapError | 0x2079e6d0e340>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2079e6b8e140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2079e6d0e300>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j8436224-mgmt-1\"; let ns = \"ocm-arohcpci01-2u2bolntenjo2h2nlf59v2siljo5e24v-cluster\"; let start = datetime(2026-11-13T13:00:30.5373611Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2079e80a8180>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x24dd696984c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
err: <*verifiers.pollObservationError | 0x24dd696984a0>{
observation: <*errors.errorString | 0x24dd688c93b0>{
s: "dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
},
interruption: <*fmt.wrapError | 0x24dd69698480>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x24dd6875c140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x24dd69698460>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x24dd6a0bc360>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x24dd696984c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6btqkp777kzc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
err: <*verifiers.pollObservationError | 0x24dd696984a0>{
observation: <*errors.errorString | 0x24dd688c93b0>{
s: "dev/westus3/ci01-j8698240-mgmt-2/ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster router=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003: expected matching summary after test start and within 8m; summaryUID=1f38a2e5-d09f-44c9-b325-c30c76059871 node=aks-userswft2-21219574-vmss000003 pass=76ef8b3f-12d4-46f4-8ae9-c7b9e29ddcd1 at=2026-11-13T07:41:52Z",
},
interruption: <*fmt.wrapError | 0x24dd69698480>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x24dd6875c140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x24dd69698460>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j8698240-mgmt-2\"; let ns = \"ocm-arohcpci01-2u26qiaeire3nkg3vu2f2lmj6h1pn6eu-cluster\"; let start = datetime(2026-11-13T07:23:35.5609003Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x24dd6a0bc360>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...full failure pattern: VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2af5093ab1a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x2af5093ab180>{
observation: <*errors.errorString | 0x2af508c1bd90>{
s: "dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x2af5093ab160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2af508aa2820>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2af5093ab140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2af509e57260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:113]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2af5093ab1a0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59t2st96lp87/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
err: <*verifiers.pollObservationError | 0x2af5093ab180>{
observation: <*errors.errorString | 0x2af508c1bd90>{
s: "dev/westus3/ci01-j2021760-mgmt-2/ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
},
interruption: <*fmt.wrapError | 0x2af5093ab160>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2af508aa2820>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2af5093ab140>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j2021760-mgmt-2\"; let ns = \"ocm-arohcpci01-2u265sbpcfhu9n3m8h8ncqtonksjshv0-cluster\"; let start = datetime(2026-11-13T06:39:36.5268067Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2af509e57260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xjngq8vfmt4g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2c25c2fb81c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xjngq8vfmt4g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j6801664-mgmt-2/ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster router=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003: expected matching summary after test start and within 8m; summaryUID=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003 pass=4dc8acd7-ce29-46fc-9f48-b487d3e910d8 at=2026-11-12T03:54:53Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xjngq8vfmt4g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j6801664-mgmt-2/ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster router=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003: expected matching summary after test start and within 8m; summaryUID=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003 pass=4dc8acd7-ce29-46fc-9f48-b487d3e910d8 at=2026-11-12T03:54:53Z",
err: <*verifiers.pollObservationError | 0x2c25c2fb81a0>{
observation: <*errors.errorString | 0x2c25c168ae40>{
s: "dev/westus3/ci01-j6801664-mgmt-2/ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster router=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003: expected matching summary after test start and within 8m; summaryUID=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003 pass=4dc8acd7-ce29-46fc-9f48-b487d3e910d8 at=2026-11-12T03:54:53Z",
},
interruption: <*fmt.wrapError | 0x2c25c2fb8180>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6801664-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster\"; let start = datetime(2026-11-12T03:39:28.9054828Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2c25c1d42140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2c25c2fb8160>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6801664-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster\"; let start = datetime(2026-11-12T03:39:28.9054828Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2c25c3ec2150>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xjngq8vfmt4g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2c25c2fb81c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xjngq8vfmt4g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j6801664-mgmt-2/ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster router=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003: expected matching summary after test start and within 8m; summaryUID=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003 pass=4dc8acd7-ce29-46fc-9f48-b487d3e910d8 at=2026-11-12T03:54:53Z
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-xjngq8vfmt4g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j6801664-mgmt-2/ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster router=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003: expected matching summary after test start and within 8m; summaryUID=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003 pass=4dc8acd7-ce29-46fc-9f48-b487d3e910d8 at=2026-11-12T03:54:53Z",
err: <*verifiers.pollObservationError | 0x2c25c2fb81a0>{
observation: <*errors.errorString | 0x2c25c168ae40>{
s: "dev/westus3/ci01-j6801664-mgmt-2/ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster router=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003: expected matching summary after test start and within 8m; summaryUID=ade633bc-72a0-4c87-8871-492106b29cd5 node=aks-userswft1-18612150-vmss000003 pass=4dc8acd7-ce29-46fc-9f48-b487d3e910d8 at=2026-11-12T03:54:53Z",
},
interruption: <*fmt.wrapError | 0x2c25c2fb8180>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6801664-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster\"; let start = datetime(2026-11-12T03:39:28.9054828Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2c25c1d42140>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2c25c2fb8160>{
msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j6801664-mgmt-2\"; let ns = \"ocm-arohcpci01-2u1ee9v293s4lc6sp3ru1j9qrot72pmv-cluster\"; let start = datetime(2026-11-12T03:39:28.9054828Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2c25c3ec2150>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubeNodeReadinessFlapping firedfull failure pattern: alert [svc] KubeNodeReadinessFlapping fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | alert | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-13T22:51:23Z Ended: 2026-11-13T22:58:23Z Severity: Sev3 Labels: alertname="KubeNodeReadinessFlapping", cluster="ci01-j0177792-mgmt-2", component="kubernetes-infrastructure", node="aks-userswft3-14254286-vmss000004", region="centralus", severity="warning" Description: The readiness status of node aks-userswft3-14254286-vmss000004 has changed 4 times in the last 15 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-12T06:50:40Z Ended: 2026-11-12T06:57:40Z Severity: Sev3 Labels: alertname="KubeNodeReadinessFlapping", cluster="ci01-j4295552-mgmt-1", component="kubernetes-infrastructure", node="aks-userswft3-29188765-vmss000003", region="centralus", severity="warning" Description: The readiness status of node aks-userswft3-29188765-vmss000003 has changed 4 times in the last 15 minutes. Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:159]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:159]: Cluster provisioning failed Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
SamplesAvailable condition must be Truefull failure pattern: SamplesAvailable condition must be True Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.000s.
HCP resource requirements were not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with:
SamplesAvailable condition must be True
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.000s.
HCP resource requirements were not populated in time
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with:
SamplesAvailable condition must be True
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] ServiceCPUDrift firedfull failure pattern: alert [svc] ServiceCPUDrift fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | alert | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-16T17:58:45Z Ended: 2026-11-16T18:10:47Z Severity: Sev3 Labels: alertname="ServiceCPUDrift", cluster="ci01-j6261632-mgmt-2", component="service-memory", container="kube-events", namespace="monitoring", pod="kube-events-cb4677688-cc75w", severity="warning", team="hcp-sl" Description: Container kube-events in pod kube-events-cb4677688-cc75w (namespace monitoring) on cluster ci01-j6261632-mgmt-2 has averaged 142.2% of its CPU request over the last 30 minutes. This may indicate workload growth that requires right-sizing the request in config.yaml. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-11-12T16:15:22Z Ended: 2026-11-12T16:21:22Z Severity: Sev3 Labels: alertname="ServiceCPUDrift", cluster="ci01-j5531776-mgmt-1", component="service-memory", container="kube-events", namespace="monitoring", pod="kube-events-7fb9b66b55-xhs6n", severity="warning", team="hcp-sl" Description: Container kube-events in pod kube-events-7fb9b66b55-xhs6n (namespace monitoring) on cluster ci01-j5531776-mgmt-1 has averaged 122.5% of its CPU request over the last 30 minutes. This may indicate workload growth that requires right-sizing the request in config.yaml. Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"full failure pattern: cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsNov 10: 1 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:228]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:224 with:
cluster delayed-rbac-jl52d56xkk54/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:228]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:224 with:
cluster delayed-rbac-jl52d56xkk54/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-56f7rr2xdxmt/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-56f7rr2xdxmt/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: FailedContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to roll out Helm release: cluster reachability check failed: kubernetes cluster unreachable: the server is currently unable to handle the requestfull failure pattern: failed to roll out Helm release: cluster reachability check failed: kubernetes cluster unreachable: the server is currently unable to handle the request Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | provision | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)time=2026-11-13T09:16:08.194Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-11-13T09:16:08.646Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 values="map[cosmosDB:map[containerMaxScale:19000 containerName:Manifests-MC-2 databaseName:arohcpci01-rp-j8177024 endpoint:https://arohcpci01-rp-j8177024.documents.azure.com:443/] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-khc1n5y0/pipeline@sha256:4758c880d0b071cd7e8652b79fcfcc6e27fad81b2ff72209c489c48366575889 replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:260m memory:224Mi]] zoneCount:3] managementCluster:/providers/microsoft.redhatopenshift/stamps/2/managementclusters/default monitoringApiGroup:monitoring.coreos.com pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-khc1n5y0/pipeline:pull workloadIdentityClientId:a58e8353-bf48-48af-947d-f4185e107d47 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] schedulingRole:infra serviceAccount:map[name:kube-applier workloadIdentityClientId:f64437cb-7622-43bc-bf8e-4b484947f086 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]" time=2026-11-13T09:16:08.646Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 time=2026-11-13T09:16:09.094Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 namespace=kube-applier time=2026-11-13T09:16:09.370Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-11-13T09:16:09.370Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 time=2026-11-13T09:16:09.473Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 time=2026-11-13T09:16:29.572Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 err="cluster reachability check failed: kubernetes cluster unreachable: the server is currently unable to handle the request" time=2026-11-13T09:16:29.572Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.KubeApplier resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: cluster reachability check failed: kubernetes cluster unreachable: the server is currently unable to handle the request" time=2026-11-12T23:49:52.123Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull description="Step acrpull\n Kind: Helm\n" time=2026-11-12T23:49:52.522Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:267697be88449eb020f149c98e0bff9dc4d9d2acdbc2cb993209030fc9eb6972 monitoringApiGroup:monitoring.coreos.com namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-11-12T23:49:52.522Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-11-12T23:49:52.879Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull namespace=acrpull time=2026-11-12T23:49:53.036Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull dryRun=false time=2026-11-12T23:49:53.036Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-11-12T23:49:53.133Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-11-12T23:49:53.157Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="cluster reachability check failed: kubernetes cluster unreachable: the server is currently unable to handle the request" time=2026-11-12T23:49:53.157Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: cluster reachability check failed: kubernetes cluster unreachable: the server is currently unable to handle the request" Contributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to wait for first cluster "<cluster>" to complete creation (timeout '30.000000' minutes)full failure pattern: failed to wait for first cluster "<cluster>" to complete creation (timeout '30.000000' minutes) Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 2 | 1.04%2 of 192 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsNov 10: 1 · Nov 11: 1 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:132]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:132]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
{}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 2 | 7.69%2 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 7 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:99]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x17b1600e4de0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-psp6ghkd2r7p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"name": "cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"status": "Failed",
"startTime": "2026-11-12T19:48:15.197021893Z",
"endTime": "2026-11-12T20:17:16.216474731Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 25m13s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
correlation ID: 588080de-4376-4b27-928a-03d587d57b2d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:99]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x17b1600e4de0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-psp6ghkd2r7p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"name": "cca8b8cc-2343-49ae-888f-3d4f2b5753a5",
"status": "Failed",
"startTime": "2026-11-12T19:48:15.197021893Z",
"endTime": "2026-11-12T20:17:16.216474731Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 25m13s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cca8b8cc-2343-49ae-888f-3d4f2b5753a5
correlation ID: 588080de-4376-4b27-928a-03d587d57b2d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-rqw6dbbkwpq8/cluster-candidate-5-1-25sx5t should provision
Unexpected error:
<*fmt.wrapError | 0x2f3a166b8800>:
failed to create HCP cluster cluster-candidate-5-1-25sx5t: failed waiting for cluster="cluster-candidate-5-1-25sx5t" in resourcegroup="rg-candidate-5-1-rqw6dbbkwpq8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489",
"name": "0ad08df1-5222-482b-8e92-25a4d0649489",
"status": "Failed",
"startTime": "2026-11-12T19:48:14.366989654Z",
"endTime": "2026-11-12T20:17:15.941327806Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 23m10s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
correlation ID: f77ffdcc-6db4-4939-bd7d-f626c9d73793
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-rqw6dbbkwpq8/cluster-candidate-5-1-25sx5t should provision
Unexpected error:
<*fmt.wrapError | 0x2f3a166b8800>:
failed to create HCP cluster cluster-candidate-5-1-25sx5t: failed waiting for cluster="cluster-candidate-5-1-25sx5t" in resourcegroup="rg-candidate-5-1-rqw6dbbkwpq8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489",
"name": "0ad08df1-5222-482b-8e92-25a4d0649489",
"status": "Failed",
"startTime": "2026-11-12T19:48:14.366989654Z",
"endTime": "2026-11-12T20:17:15.941327806Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 23m10s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0ad08df1-5222-482b-8e92-25a4d0649489
correlation ID: f77ffdcc-6db4-4939-bd7d-f626c9d73793
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x28e2f0a88a80>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nfkbd8xd9kxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5",
"name": "324ccd68-ef42-41e9-a980-fd643a150df5",
"status": "Failed",
"startTime": "2026-11-12T17:11:50.950828556Z",
"endTime": "2026-11-12T17:40:56.295707627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 20m27s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
correlation ID: a04ff94c-430d-45c3-978d-87b26669fcf4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:109]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x28e2f0a88a80>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nfkbd8xd9kxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5",
"name": "324ccd68-ef42-41e9-a980-fd643a150df5",
"status": "Failed",
"startTime": "2026-11-12T17:11:50.950828556Z",
"endTime": "2026-11-12T17:40:56.295707627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.42 is Partial (want Completed), started 20m27s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/324ccd68-ef42-41e9-a980-fd643a150df5
correlation ID: a04ff94c-430d-45c3-978d-87b26669fcf4
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 2 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:100]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
<*fmt.wrapError | 0x1b4614d815e0>:
failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pk4g7jxvwskz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"name": "5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"status": "Failed",
"startTime": "2026-11-12T17:11:52.373940715Z",
"endTime": "2026-11-12T17:40:56.797994555Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 22m16s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
correlation ID: ed91409f-3f3e-40f0-af6f-e1efc0a6cad7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:100]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
<*fmt.wrapError | 0x1b4614d815e0>:
failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-pk4g7jxvwskz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"name": "5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0",
"status": "Failed",
"startTime": "2026-11-12T17:11:52.373940715Z",
"endTime": "2026-11-12T17:40:56.797994555Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.22 is Partial (want Completed), started 22m16s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e76d713-722e-4ddb-85a3-0b1ccd6fc4a0
correlation ID: ed91409f-3f3e-40f0-af6f-e1efc0a6cad7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-9qmh46" with version 5.0.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3c857efdd5e0>:
failed to create HCP cluster cluster-zstream-5-0-9qmh46: failed waiting for cluster="cluster-zstream-5-0-9qmh46" in resourcegroup="rg-zstream-upgrade-5-0-2dkdlwm4dztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e",
"name": "74ae707c-8d62-4995-b262-0153bb8bf32e",
"status": "Failed",
"startTime": "2026-11-12T17:11:53.237843141Z",
"endTime": "2026-11-12T17:40:56.804735597Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0 is Partial (want Completed), started 22m14s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
correlation ID: b786b9a5-3d9a-4d46-bae8-4f64d6429da9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-9qmh46" with version 5.0.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3c857efdd5e0>:
failed to create HCP cluster cluster-zstream-5-0-9qmh46: failed waiting for cluster="cluster-zstream-5-0-9qmh46" in resourcegroup="rg-zstream-upgrade-5-0-2dkdlwm4dztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e",
"name": "74ae707c-8d62-4995-b262-0153bb8bf32e",
"status": "Failed",
"startTime": "2026-11-12T17:11:53.237843141Z",
"endTime": "2026-11-12T17:40:56.804735597Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0 is Partial (want Completed), started 22m14s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74ae707c-8d62-4995-b262-0153bb8bf32e
correlation ID: b786b9a5-3d9a-4d46-bae8-4f64d6429da9
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x17a04ae8aee0>:
failed to create HCP cluster np-dg-minor-89bskt: failed waiting for cluster="np-dg-minor-89bskt" in resourcegroup="rg-np-dg-minor-89bskt-sv5nkfqrjg8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4",
"name": "7966bc18-a5b0-4651-b41c-fc426eb273c4",
"status": "Failed",
"startTime": "2026-11-14T20:41:04.80974955Z",
"endTime": "2026-11-14T21:10:07.827174067Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
correlation ID: 9c07c544-07d2-4418-852b-26b1b9c7f5ee
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x17a04ae8aee0>:
failed to create HCP cluster np-dg-minor-89bskt: failed waiting for cluster="np-dg-minor-89bskt" in resourcegroup="rg-np-dg-minor-89bskt-sv5nkfqrjg8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4",
"name": "7966bc18-a5b0-4651-b41c-fc426eb273c4",
"status": "Failed",
"startTime": "2026-11-14T20:41:04.80974955Z",
"endTime": "2026-11-14T21:10:07.827174067Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7966bc18-a5b0-4651-b41c-fc426eb273c4
correlation ID: 9c07c544-07d2-4418-852b-26b1b9c7f5ee
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining res...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x1a1012fe330>:
resource group "feature-aggregation-b6fvjljtlcvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-b6fvjljtlcvs" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c",
"name": "e2b882db-48de-4ff2-9f50-4222b9955e6c",
"status": "Failed",
"startTime": "2026-11-16T02:36:07.442208668Z",
"endTime": "2026-11-16T03:00:10.537191563Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
correlation ID: 9b1360be-096b-4387-8316-508cc19a90f1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:400]: Unexpected error:
<*errors.joinError | 0x1a1012fe330>:
resource group "feature-aggregation-b6fvjljtlcvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-b6fvjljtlcvs" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c",
"name": "e2b882db-48de-4ff2-9f50-4222b9955e6c",
"status": "Failed",
"startTime": "2026-11-16T02:36:07.442208668Z",
"endTime": "2026-11-16T03:00:10.537191563Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e2b882db-48de-4ff2-9f50-4222b9955e6c
correlation ID: 9b1360be-096b-4387-8316-508cc19a90f1
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-m7c8jjt7kz5k/cluster-candidate-5-1-zwr77b should provision
Unexpected error:
<*fmt.wrapError | 0x317411260>:
failed to create HCP cluster cluster-candidate-5-1-zwr77b: failed waiting for cluster="cluster-candidate-5-1-zwr77b" in resourcegroup="rg-candidate-5-1-m7c8jjt7kz5k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93",
"name": "d222ac87-ad5a-4334-8130-3c30fedb6e93",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.805632783Z",
"endTime": "2026-11-12T17:40:46.399524896Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 20m19s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
correlation ID: 6a614cae-828b-4e97-ac1e-2aac8b7ca153
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-1-m7c8jjt7kz5k/cluster-candidate-5-1-zwr77b should provision
Unexpected error:
<*fmt.wrapError | 0x317411260>:
failed to create HCP cluster cluster-candidate-5-1-zwr77b: failed waiting for cluster="cluster-candidate-5-1-zwr77b" in resourcegroup="rg-candidate-5-1-m7c8jjt7kz5k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93",
"name": "d222ac87-ad5a-4334-8130-3c30fedb6e93",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.805632783Z",
"endTime": "2026-11-12T17:40:46.399524896Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.2 is Partial (want Completed), started 20m19s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d222ac87-ad5a-4334-8130-3c30fedb6e93
correlation ID: 6a614cae-828b-4e97-ac1e-2aac8b7ca153
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x26c329a4a8e0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rk97m722fgxr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"name": "6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"status": "Failed",
"startTime": "2026-11-12T17:14:14.481992445Z",
"endTime": "2026-11-12T17:43:16.629667554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, packageserver, cluster-network-operator, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, ingress-operator, redhat-operators-catalog, openshift-controller-manager, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog, catalog-operator, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
correlation ID: c03cdfce-2433-48c4-b0bf-dd2331b70f7a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x26c329a4a8e0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rk97m722fgxr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"name": "6f9079cf-1a15-4376-9f0a-f5cce42aa4b4",
"status": "Failed",
"startTime": "2026-11-12T17:14:14.481992445Z",
"endTime": "2026-11-12T17:43:16.629667554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-marketplace-catalog, packageserver, cluster-network-operator, cluster-node-tuning-operator, dns-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, ingress-operator, redhat-operators-catalog, openshift-controller-manager, cluster-storage-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog, catalog-operator, cluster-version-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f9079cf-1a15-4376-9f0a-f5cce42aa4b4
correlation ID: c03cdfce-2433-48c4-b0bf-dd2331b70f7a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-rnvnjl with version 4.21.42
Unexpected error:
<*fmt.wrapError | 0x33618be5f5c0>:
failed to create HCP cluster np-version-upgrade-cluster-rnvnjl: failed waiting for cluster="np-version-upgrade-cluster-rnvnjl" in resourcegroup="rg-np-version-upgrade-rnvnjl-khbkdtxdkfdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a",
"name": "3f68db84-4a8d-4bf7-9849-3e7db026805a",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.409592604Z",
"endTime": "2026-11-12T17:40:46.473882071Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
correlation ID: 2c6cdab0-a3af-4588-8f80-5ebc352de1b7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-rnvnjl with version 4.21.42
Unexpected error:
<*fmt.wrapError | 0x33618be5f5c0>:
failed to create HCP cluster np-version-upgrade-cluster-rnvnjl: failed waiting for cluster="np-version-upgrade-cluster-rnvnjl" in resourcegroup="rg-np-version-upgrade-rnvnjl-khbkdtxdkfdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a",
"name": "3f68db84-4a8d-4bf7-9849-3e7db026805a",
"status": "Failed",
"startTime": "2026-11-12T17:11:41.409592604Z",
"endTime": "2026-11-12T17:40:46.473882071Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3f68db84-4a8d-4bf7-9849-3e7db026805a
correlation ID: 2c6cdab0-a3af-4588-8f80-5ebc352de1b7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: RevokeCredentialsAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:223]: failed to revoke admin credentials for cluster "admin-cred-lifecycle-vgkc2m"
Unexpected error:
<*fmt.wrapErrors | 0x41bbce44ea0>:
failed waiting for hcpCluster="admin-cred-lifecycle-vgkc2m" in resourcegroup="admin-credential-lifecycle-test-bgcs6sp9q9hf" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-vgkc2m\" in resourcegroup=\"admin-credential-lifecycle-test-bgcs6sp9q9hf\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x41bbbe36b00>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:223]: failed to revoke admin credentials for cluster "admin-cred-lifecycle-vgkc2m"
Unexpected error:
<*fmt.wrapErrors | 0x41bbce44ea0>:
failed waiting for hcpCluster="admin-cred-lifecycle-vgkc2m" in resourcegroup="admin-credential-lifecycle-test-bgcs6sp9q9hf" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-vgkc2m\" in resourcegroup=\"admin-credential-lifecycle-test-bgcs6sp9q9hf\" to finish revoking creds, caused by: timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x41bbbe36b00>{
s: "timeout '15.000000' minutes exceeded during RevokeCredentialsAndWait for cluster admin-cred-lifecycle-vgkc2m in resource group admin-credential-lifecycle-test-bgcs6sp9q9hf",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
Unauthorizedfull failure pattern: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:236]: Timed out after 300.001s.
Expected success, but got an error:
<*errors.StatusError | 0x38a7b05b720>:
Unauthorized
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:236]: Timed out after 300.001s.
Expected success, but got an error:
<*errors.StatusError | 0x38a7b05b720>:
Unauthorized
{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod <namespace>/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the r...full failure pattern: VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod <namespace>/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the request from succeeding (get pod <pod>)Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 1 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-20-tc28nz"
Unexpected error:
<*errors.joinError | 0x227e60b84150>:
VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the request from succeeding (get pods e2e-csi-pod)
{
errs: [
<*fmt.wrapError | 0x227e601b6be0>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*fmt.wrapError | 0x227e6113c8c0>{
msg: "failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*errors.StatusError | 0x227e60a46dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
Reason: "InternalError",
Details: {
Name: "e2e-csi-pod",
Group: "",
Kind: "pods",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "unknown",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-20-tc28nz"
Unexpected error:
<*errors.joinError | 0x227e60b84150>:
VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server ("unknown") has prevented the request from succeeding (get pods e2e-csi-pod)
{
errs: [
<*fmt.wrapError | 0x227e601b6be0>{
msg: "VerifyCSIPersistentVolume(file.csi.azure.com) failed: failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*fmt.wrapError | 0x227e6113c8c0>{
msg: "failed to get logs for pod e2e-csi-file-47wdl/e2e-csi-pod after it succeeded: an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
err: <*errors.StatusError | 0x227e60a46dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "an error on the server (\"unknown\") has prevented the request from succeeding (get pods e2e-csi-pod)",
Reason: "InternalError",
Details: {
Name: "e2e-csi-pod",
Group: "",
Kind: "pods",
UID: "",
Causes: [
{
Type: "UnexpectedServerResponse",
Message: "unknown",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-lftmdz-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node cluster-np-lftmdz-<id> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-lftmdz-<id>": nodes "cluster-np-lftmdz-<id>" is forbidden Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-lftmdz"
Unexpected error:
<*errors.joinError | 0x2c80b1a3b560>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-lftmdz-ghqg5-5jcqq": nodes "cluster-np-lftmdz-ghqg5-5jcqq" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2c80b1940940>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940920>{
msg: "failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940900>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2c80b186d7c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-lftmdz-ghqg5-5jcqq",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:299]: failed to verify cluster operators on cluster "cluster-candidate-4-22-lftmdz"
Unexpected error:
<*errors.joinError | 0x2c80b1a3b560>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "cluster-np-lftmdz-ghqg5-5jcqq": nodes "cluster-np-lftmdz-ghqg5-5jcqq" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2c80b1940940>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940920>{
msg: "failed to label node cluster-np-lftmdz-ghqg5-5jcqq egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2c80b1940900>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"cluster-np-lftmdz-ghqg5-5jcqq\": nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2c80b186d7c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"cluster-np-lftmdz-ghqg5-5jcqq\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "cluster-np-lftmdz-ghqg5-5jcqq",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp...full failure pattern: VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-4wrz4s"
Unexpected error:
<*fmt.wrapError | 0x244b4deba960>:
VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-4fxqjqb"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-4fxqjqb\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase=\"Running\" reason=\"\"",
err: <*verifiers.pollObservationError | 0x244b4deea6c0>{
observation: <*errors.errorString | 0x244b4fb06d40>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net",
},
interruption: <*fmt.wrapError | 0x244b4deea660>{
msg: "console login redirect: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-4wrz4s"
Unexpected error:
<*fmt.wrapError | 0x244b4deba960>:
VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-4fxqjqb"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-fsh97 container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.003s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-4fxqjqb\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-fsh97 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-fsh97 container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-fsh97 phase=\"Running\" reason=\"\"",
err: <*verifiers.pollObservationError | 0x244b4deea6c0>{
observation: <*errors.errorString | 0x244b4fb06d40>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.kl99l.westus3.aroapp-hcp.azure-test.net",
},
interruption: <*fmt.wrapError | 0x244b4deea660>{
msg: "console login redirect: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp...full failure pattern: VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-97dm56"
Unexpected error:
<*fmt.wrapError | 0x1dbf7ed98e40>:
VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-fn9vpbv"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-fn9vpbv\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x1dbf809b21d0>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:257]: failed to verify Console login for cluster "cluster-candidate-5-1-97dm56"
Unexpected error:
<*fmt.wrapError | 0x1dbf7ed98e40>:
VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net
Dex namespace="console-fn9vpbv"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition="ContainersReady" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Initialized" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="PodScheduled" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr condition="Ready" status="True" reason=""; Pod/dex-66b9c7df45-d4xbr container="dex" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase="Running" reason=""
{
msg: "VerifyConsoleLogin timed out after 15m0.004s: console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net\nDex namespace=\"console-fn9vpbv\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-66b9c7df45-d4xbr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-66b9c7df45-d4xbr container=\"dex\" ready=true restarts=0; Pod/dex-66b9c7df45-d4xbr phase=\"Running\" reason=\"\"",
err: <*errors.errorString | 0x1dbf809b21d0>{
s: "console login expected Dex password page, observed HTTP 200 from host console-openshift-console.apps.aro.cluster.xfh7r.westus3.aroapp-hcp.azure-test.net",
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in name...full failure pattern: polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors: Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:346]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpull9vvdflkz2tvn.azurecr.io
Unexpected error:
<*fmt.wrapError | 0x1d5b57c7d1c0>:
image pull not confirmed; cause undetermined (insufficient evidence)
polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
- pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d
{
msg: "image pull not confirmed; cause undetermined (insufficient evidence)\npolling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*fmt.wrapError | 0x1d5b57c7d180>{
msg: "VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*verifiers.pollObservationError | 0x1d5b57c7d040>{
observation: <*errors.errorString | 0x1d5b59f3bdd0>{
s: "expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
},
interruption: <*fmt.wrapError | 0x1d5b57c7d020>{
msg: "failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:346]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpull9vvdflkz2tvn.azurecr.io
Unexpected error:
<*fmt.wrapError | 0x1d5b57c7d1c0>:
image pull not confirmed; cause undetermined (insufficient evidence)
polling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:
- pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d
{
msg: "image pull not confirmed; cause undetermined (insufficient evidence)\npolling observation: VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*fmt.wrapError | 0x1d5b57c7d180>{
msg: "VerifyImagePulled(acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal) timed out after 2m0.004s: expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
err: <*verifiers.pollObservationError | 0x1d5b57c7d040>{
observation: <*errors.errorString | 0x1d5b59f3bdd0>{
s: "expected image from acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal in namespace acr-pull-test-day2: image pull errors:\n - pod acr-pull-test container acr-pull-test: ErrImagePull - unable to pull image or OCI artifact: pull image err: initializing source docker://acrpull9vvdflkz2tvn.azurecr.io/ubi-minimal:latest: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3302f063-1f98-4781-a1db-ceb19c7649a2; artifact err: image reference: get manifest from ref: create image source: unable to retrieve auth token: invalid username/password: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3f4f6643-530b-4fdb-bf7d-5196d40b653d",
},
interruption: <*fmt.wrapError | 0x1d5b57c7d020>{
msg: "failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 3.85%1 of 26 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:153]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x19f475ced140>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c96mksh6wtzx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-c96mksh6wtzx\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x19f474e59970>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:153]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x19f475ced140>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c96mksh6wtzx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-c96mksh6wtzx\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x19f474e59970>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-c96mksh6wtzx",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 90.91% | 11 | stage/parallel | Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor | |
| 90.91% | 11 | stage/parallel | Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | DEV, PROD | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:149]: failed to verify cluster "cp-ystream-upgrade-5-0-jgz7z5" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x3e96a1c86db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e96a1d21c80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e96a1d21c60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e96a28e4820>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:149]: failed to verify cluster "cp-ystream-upgrade-5-0-jgz7z5" is viable before upgrade
Unexpected error:
<*errors.joinError | 0x3e96a1c86db0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e96a1d21c80>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e96a1d21c60>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e96a28e4820>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 0 · Nov 14: 1 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | PROD | ||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x2b922c8b8090>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2b922c896040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2b922be8e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2b922c8960a0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896080>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2b922be8efa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x2b922c8b8090>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2b922c896040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2b922be8e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2b922c8960a0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2b922c896080>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2b922be8efa0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 90.00% | 10 | prod/parallel | Customer should be able to get a specific HCP Operator Identity Role Set by version | |
| 90.00% | 10 | prod/parallel | Customer should be able to list available HCP Operator Identity Role Sets and validate response content | |
| 90.00% | 10 | prod/parallel | Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 3 | 7.14%3 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 3 · Nov 12: 0 · Nov 13: 6 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7"
Unexpected error:
<*fmt.wrapError | 0x3eb94d03ed60>:
failed to create NodePool np-k5wcs7: failed waiting for nodepool="np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7" in resourcegroup="rg-candidate-4-23-fvhwkprkl5hk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926",
"name": "edad2cce-0c27-4141-af66-5421d0367926",
"status": "Failed",
"startTime": "2026-11-13T06:43:01.877199954Z",
"endTime": "2026-11-13T07:02:05.836906321Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: Creating: virtualmachine creating or updating\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
correlation ID: b6db496e-1ecf-4f45-98bf-19f14c9919e4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7"
Unexpected error:
<*fmt.wrapError | 0x3eb94d03ed60>:
failed to create NodePool np-k5wcs7: failed waiting for nodepool="np-k5wcs7" for cluster "cluster-candidate-4-23-k5wcs7" in resourcegroup="rg-candidate-4-23-fvhwkprkl5hk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926",
"name": "edad2cce-0c27-4141-af66-5421d0367926",
"status": "Failed",
"startTime": "2026-11-13T06:43:01.877199954Z",
"endTime": "2026-11-13T07:02:05.836906321Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-k5wcs7-5xqx2-5rjs2: Creating: virtualmachine creating or updating\nMachine cluster-np-k5wcs7-5xqx2-t6n2g: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edad2cce-0c27-4141-af66-5421d0367926
correlation ID: b6db496e-1ecf-4f45-98bf-19f14c9919e4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x2c3a6de09320>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-pvc2kk" in resourcegroup="rg-np-noedge-pvc2kk-f5hxq2mnnx2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8",
"name": "f41964ee-de97-4466-b5e8-feb618fc82a8",
"status": "Failed",
"startTime": "2026-11-13T07:19:26.464221032Z",
"endTime": "2026-11-13T07:38:31.657256324Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-4wlgb-528bt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-4wlgb-528bt: Creating: virtualmachine creating or updating\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
correlation ID: e2a37064-75a7-4e62-90a9-71bfa7da63c5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.45
Unexpected error:
<*fmt.wrapError | 0x2c3a6de09320>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-pvc2kk" in resourcegroup="rg-np-noedge-pvc2kk-f5hxq2mnnx2r" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8",
"name": "f41964ee-de97-4466-b5e8-feb618fc82a8",
"status": "Failed",
"startTime": "2026-11-13T07:19:26.464221032Z",
"endTime": "2026-11-13T07:38:31.657256324Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npnoedge-4-20-4wlgb-528bt: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npnoedge-4-20-4wlgb-528bt: Creating: virtualmachine creating or updating\nMachine cluster-npnoedge-4-20-4wlgb-sjlb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f41964ee-de97-4466-b5e8-feb618fc82a8
correlation ID: e2a37064-75a7-4e62-90a9-71bfa7da63c5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz"
Unexpected error:
<*fmt.wrapError | 0x17145c623cc0>:
failed to create NodePool np-pwhdqz: failed waiting for nodepool="np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz" in resourcegroup="rg-candidate-5-1-z27fm55sbdhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd",
"name": "8994658d-0314-4930-b123-77a9aa8d6ddd",
"status": "Failed",
"startTime": "2026-11-13T06:47:16.516948762Z",
"endTime": "2026-11-13T07:06:16.339079825Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-pwhdqz-nrnj4-nllg8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-pwhdqz-nrnj4-nllg8: Creating: virtualmachine creating or updating\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
correlation ID: 67c21db5-2743-4fb5-a2e0-d37a965dd5f1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz"
Unexpected error:
<*fmt.wrapError | 0x17145c623cc0>:
failed to create NodePool np-pwhdqz: failed waiting for nodepool="np-pwhdqz" for cluster "cluster-candidate-5-1-pwhdqz" in resourcegroup="rg-candidate-5-1-z27fm55sbdhh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd",
"name": "8994658d-0314-4930-b123-77a9aa8d6ddd",
"status": "Failed",
"startTime": "2026-11-13T06:47:16.516948762Z",
"endTime": "2026-11-13T07:06:16.339079825Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-pwhdqz-nrnj4-nllg8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-pwhdqz-nrnj4-nllg8: Creating: virtualmachine creating or updating\nMachine cluster-np-pwhdqz-nrnj4-xc9cl: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8994658d-0314-4930-b123-77a9aa8d6ddd
correlation ID: 67c21db5-2743-4fb5-a2e0-d37a965dd5f1
client request ID: unavailable
occurredContributing tests (8)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 2 | 4.76%2 of 42 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsNov 10: 1 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 1 · Nov 16: 0 · Nov 17: 0 | DEV, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:201]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x1883079cea80>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x188307f50040>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x188307f50020>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x18830755e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1883079de2c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1883079de2a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1883073ec960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:201]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x1883079cea80>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x188307f50040>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x188307f50020>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x18830755e8c0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x1883079de2c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x1883079de2a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x1883073ec960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x5ec36ffd968>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5ec37183140>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5ec37183120>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5ec36a47680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5ec3701ed60>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5ec3701ed40>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5ec36c2fe00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:106]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x5ec36ffd968>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5ec37183140>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5ec37183120>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5ec36a47680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5ec3701ed60>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5ec3701ed40>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5ec36c2fe00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 4.76%2 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsNov 10: 0 · Nov 11: 3 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1f4a465481a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gvns5xqr82fn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"name": "568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"status": "Failed",
"startTime": "2026-11-13T06:13:10.507550785Z",
"endTime": "2026-11-13T06:32:15.73883621Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-9sd6w-4wd6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-9sd6w-hng7m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-9sd6w-4wd6d: Creating: virtualmachine creating or updating\nMachine cluster-np-1-9sd6w-hng7m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
correlation ID: 218fb2af-fc43-4aa1-8295-19449adca4e0
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1f4a465481a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-gvns5xqr82fn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"name": "568295c4-1c9f-43c5-b4ff-8e6dc0c304b6",
"status": "Failed",
"startTime": "2026-11-13T06:13:10.507550785Z",
"endTime": "2026-11-13T06:32:15.73883621Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-9sd6w-4wd6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-np-1-9sd6w-hng7m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-9sd6w-4wd6d: Creating: virtualmachine creating or updating\nMachine cluster-np-1-9sd6w-hng7m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/568295c4-1c9f-43c5-b4ff-8e6dc0c304b6
correlation ID: 218fb2af-fc43-4aa1-8295-19449adca4e0
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3b86480f0000>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ll4kkgrplwxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613",
"name": "0d38b567-2f27-471a-84ef-558a5c67e613",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.72317124Z",
"endTime": "2026-11-11T01:32:09.867050757Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-n2m78-xpsvx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-n2m78-xpsvx: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
correlation ID: 16057b98-aa78-45ee-a357-4e003f509084
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3b86480f0000>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-ll4kkgrplwxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613",
"name": "0d38b567-2f27-471a-84ef-558a5c67e613",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.72317124Z",
"endTime": "2026-11-11T01:32:09.867050757Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-n2m78-xpsvx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-n2m78-xpsvx: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0d38b567-2f27-471a-84ef-558a5c67e613
correlation ID: 16057b98-aa78-45ee-a357-4e003f509084
client request ID: unavailablefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x398924e8c5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rb5c9dvb42gv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84",
"name": "21d8f1e5-67ed-479d-8eac-deaac258db84",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.698291582Z",
"endTime": "2026-11-11T01:32:09.741837947Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-8jqlq-kjgfc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-8jqlq-kjgfc: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
correlation ID: 5eca5994-d282-425a-909e-85f0c80c01f6
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x398924e8c5e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-rb5c9dvb42gv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84",
"name": "21d8f1e5-67ed-479d-8eac-deaac258db84",
"status": "Failed",
"startTime": "2026-11-11T01:13:06.698291582Z",
"endTime": "2026-11-11T01:32:09.741837947Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-8jqlq-kjgfc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-8jqlq-kjgfc: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21d8f1e5-67ed-479d-8eac-deaac258db84
correlation ID: 5eca5994-d282-425a-909e-85f0c80c01f6
client request ID: unavailableContributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 2 | 4.76%2 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 2 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-c2rs2z" is viable
Unexpected error:
<*errors.joinError | 0x2d2951054f60>:
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d2950dcc760>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d2950dcc720>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d29504c3180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-20-c2rs2z" is viable
Unexpected error:
<*errors.joinError | 0x2d2951054f60>:
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d2950dcc760>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d2950dcc720>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d29504c3180>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:105]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x3e1fd491ab40>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e1fd3cde060>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd3cde040>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3a44e60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3e1fd4123c80>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd4123c60>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3e07540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:105]: failed to verify HCP cluster viability for patch-name test
Unexpected error:
<*errors.joinError | 0x3e1fd491ab40>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3e1fd3cde060>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd3cde040>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3a44e60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3e1fd4123c80>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3e1fd4123c60>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3e1fd3e07540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.76%2 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 2 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x38b8c17a6b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8h7cslrzs4d4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"name": "cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"status": "Failed",
"startTime": "2026-11-13T07:22:34.538735812Z",
"endTime": "2026-11-13T07:41:41.602384099Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-lv9kw-7q9mn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-lv9kw-7q9mn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
correlation ID: 9af047e6-a985-412b-a5f1-08a259c865d2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x38b8c17a6b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-8h7cslrzs4d4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"name": "cdb95524-9aa2-40e8-bf6e-57fa01afb212",
"status": "Failed",
"startTime": "2026-11-13T07:22:34.538735812Z",
"endTime": "2026-11-13T07:41:41.602384099Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-lv9kw-7q9mn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-lv9kw-7q9mn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-7q9mn to exist\nMachine cluster-np-1-lv9kw-dxb7m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8h7cslrzs4d4--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-lv9kw-dxb7m to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cdb95524-9aa2-40e8-bf6e-57fa01afb212
correlation ID: 9af047e6-a985-412b-a5f1-08a259c865d2
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1bcb38aea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-ccvxrx6srd7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"name": "ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"status": "Failed",
"startTime": "2026-11-13T03:38:48.906568999Z",
"endTime": "2026-11-13T03:57:50.268469646Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-x596d-7l8cz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-x596d-7l8cz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
correlation ID: 975c1ede-92c5-42fb-b310-1b83c34bc7da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1bcb38aea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-ccvxrx6srd7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"name": "ab4ca645-241b-470a-b8d9-4428a8b09f8a",
"status": "Failed",
"startTime": "2026-11-13T03:38:48.906568999Z",
"endTime": "2026-11-13T03:57:50.268469646Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-x596d-7l8cz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-x596d-7l8cz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-7l8cz to exist\nMachine cluster-np-1-x596d-h2f8w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-ccvxrx6srd7z--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-x596d-h2f8w to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab4ca645-241b-470a-b8d9-4428a8b09f8a
correlation ID: 975c1ede-92c5-42fb-b310-1b83c34bc7da
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 70 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2b3cc0e02480>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-ntbnht6v2z5z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: f81e3479-b209-4c6e-aba4-7e18ebafd0a6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2b3cc0e02480>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-ntbnht6v2z5z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-ntbnht6v2z5z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: f81e3479-b209-4c6e-aba4-7e18ebafd0a6
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x2e4d7a8d39c0>:
failed to create HCP cluster des-encrypt: failed starting cluster creation "des-encrypt" in resourcegroup="des-encrypt-jtt2vbmjr52z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
correlation ID: 36af9c36-44f3-4b61-9fd9-9d1d6dda7a66
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x2e4d7a8d39c0>:
failed to create HCP cluster des-encrypt: failed starting cluster creation "des-encrypt" in resourcegroup="des-encrypt-jtt2vbmjr52z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/des-encrypt-jtt2vbmjr52z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/des-encrypt
correlation ID: 36af9c36-44f3-4b61-9fd9-9d1d6dda7a66
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x13d76780a900>:
failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-zcw7mckfcbxs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
correlation ID: 65af53cb-dfeb-48dd-a0ae-0cb96b88dc93
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x13d76780a900>:
failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-zcw7mckfcbxs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'eastasia' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zcw7mckfcbxs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
correlation ID: 65af53cb-dfeb-48dd-a0ae-0cb96b88dc93
client request ID: unavailable
occurredContributing tests (70)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClu...full failure pattern: ERROR CODE: LocationNotAvailableForResourceType; detail message The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 70 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x345936a89300>:
failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-gz55ptdrthl5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
correlation ID: 5078ed5a-a408-4bdb-ab12-939f6b2d08de
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x345936a89300>:
failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-gz55ptdrthl5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gz55ptdrthl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
correlation ID: 5078ed5a-a408-4bdb-ab12-939f6b2d08de
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x1342f0a44f80>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-d52pnvslfkxx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: d28b148b-a0dd-42a6-90c0-05ebc5318c9c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x1342f0a44f80>:
failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-d52pnvslfkxx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-d52pnvslfkxx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
correlation ID: d28b148b-a0dd-42a6-90c0-05ebc5318c9c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-95k64p" with version 4.20.45 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xd8dfd2812e0>:
failed to create HCP cluster cluster-zstream-4-20-95k64p: failed starting cluster creation "cluster-zstream-4-20-95k64p" in resourcegroup="rg-zstream-upgrade-4-20-zmsrshjlzqg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
correlation ID: 7b753e11-21cc-4524-9f3d-c18e901193f4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-95k64p" with version 4.20.45 on candidate channel
Unexpected error:
<*fmt.wrapError | 0xd8dfd2812e0>:
failed to create HCP cluster cluster-zstream-4-20-95k64p: failed starting cluster creation "cluster-zstream-4-20-95k64p" in resourcegroup="rg-zstream-upgrade-4-20-zmsrshjlzqg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: LocationNotAvailableForResourceType
--------------------------------------------------------------------------------
{
"error": {
"code": "LocationNotAvailableForResourceType",
"message": "The provided location 'swedencentral' is not available for resource type 'Microsoft.RedHatOpenShift/HCPOpenShiftClusters'. List of available regions for the resource type is 'uksouth,switzerlandnorth,canadacentral,australiaeast,westeurope,centralindia,brazilsouth,eastus2,centraluseuap,eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zmsrshjlzqg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-95k64p
correlation ID: 7b753e11-21cc-4524-9f3d-c18e901193f4
client request ID: unavailable
occurredContributing tests (70)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code VMExtensionProvisioningError; detail message VM has reported a user failure when processing extension 'installTools'. Please correct the ...full failure pattern: ERROR CODE: DeploymentFailed; detail code VMExtensionProvisioningError; detail message VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28 [stdout] [stderr] : failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at <url>; provider Microsoft.Azure.Extensions Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 3 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:164]: failed to deploy authorized CIDR VM
Unexpected error:
<*fmt.wrapError | 0x30966ca6aea0>:
failed waiting for deployment "test-vm" in resourcegroup="feature-aggregation-qfjvggxrkqkk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 6df45427-1238-4ab2-891c-02cad5db8d36",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
correlation ID: 4d33796d-6c34-40a7-af77-092588cf96e7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:164]: failed to deploy authorized CIDR VM
Unexpected error:
<*fmt.wrapError | 0x30966ca6aea0>:
failed waiting for deployment "test-vm" in resourcegroup="feature-aggregation-qfjvggxrkqkk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: 6df45427-1238-4ab2-891c-02cad5db8d36",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/feature-aggregation-qfjvggxrkqkk/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072478132550270
correlation ID: 4d33796d-6c34-40a7-af77-092588cf96e7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:101]: failed to deploy test VM for fully private with KV verification
Unexpected error:
<*fmt.wrapError | 0x259ff4de4d00>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="full-priv-kv-f9598tqlfc6q" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e1a85dbd-0af5-4f1c-b2fd-7725cc39cb74",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
correlation ID: ede4f526-5bd4-4643-b427-cc65fa41cb78
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:101]: failed to deploy test VM for fully private with KV verification
Unexpected error:
<*fmt.wrapError | 0x259ff4de4d00>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="full-priv-kv-f9598tqlfc6q" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: e1a85dbd-0af5-4f1c-b2fd-7725cc39cb74",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/full-priv-kv-f9598tqlfc6q/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072472376424978
correlation ID: ede4f526-5bd4-4643-b427-cc65fa41cb78
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:92]: failed to deploy test VM
Unexpected error:
<*fmt.wrapError | 0x3985cd6a4d40>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-98tlpt4n2kvs" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: af5f1f5c-a0eb-4321-aa22-e99f3ab0afc8",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
correlation ID: b42cc832-94c9-4550-b4e8-74a1692fb2fa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:92]: failed to deploy test VM
Unexpected error:
<*fmt.wrapError | 0x3985cd6a4d40>:
failed to deploy test VM: failed waiting for deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-98tlpt4n2kvs" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: af5f1f5c-a0eb-4321-aa22-e99f3ab0afc8",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a user failure when processing extension 'installTools'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=28\\n[stdout]\\n\\n[stderr]\\n: failed to execute command: command terminated with exit status=28'. Detailed error: ''. More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-98tlpt4n2kvs/providers/Microsoft.Resources/deployments/test-vm/operationStatuses/08584072475106235709
correlation ID: b42cc832-94c9-4550-b4e8-74a1692fb2fa
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 2 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj"
Unexpected error:
<*fmt.wrapError | 0x9ce8fef8480>:
failed to create NodePool np-hw65wj: failed waiting for nodepool="np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj" in resourcegroup="rg-candidate-4-22-njg7rgmmg8l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d",
"name": "63497251-36b4-416b-a60b-757bca5b1a6d",
"status": "Failed",
"startTime": "2026-11-13T03:59:57.353246771Z",
"endTime": "2026-11-13T04:19:05.626847985Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-hw65wj-h7fm8-hlfd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-hw65wj-h7fm8-hlfd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
correlation ID: f5bd758f-09c6-4157-8de6-ff700a1dc700
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj"
Unexpected error:
<*fmt.wrapError | 0x9ce8fef8480>:
failed to create NodePool np-hw65wj: failed waiting for nodepool="np-hw65wj" for cluster "cluster-candidate-4-22-hw65wj" in resourcegroup="rg-candidate-4-22-njg7rgmmg8l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d",
"name": "63497251-36b4-416b-a60b-757bca5b1a6d",
"status": "Failed",
"startTime": "2026-11-13T03:59:57.353246771Z",
"endTime": "2026-11-13T04:19:05.626847985Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-hw65wj-h7fm8-hlfd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-hw65wj-h7fm8-hlfd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-njg7rgmmg8l9-candidate-hw65wj--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-hw65wj-h7fm8-hlfd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/63497251-36b4-416b-a60b-757bca5b1a6d
correlation ID: f5bd758f-09c6-4157-8de6-ff700a1dc700
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5"
Unexpected error:
<*fmt.wrapError | 0x3acf2e617ec0>:
failed to create NodePool np-4smnl5: failed waiting for nodepool="np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5" in resourcegroup="rg-candidate-4-23-bqlkqphqlrhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"name": "d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"status": "Failed",
"startTime": "2026-11-13T04:02:01.972677601Z",
"endTime": "2026-11-13T04:21:05.56135744Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-4smnl5-nm98c-k5msg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-4smnl5-nm98c-k5msg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
correlation ID: fc7ff880-2948-4807-a4fc-dcd90799df98
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5"
Unexpected error:
<*fmt.wrapError | 0x3acf2e617ec0>:
failed to create NodePool np-4smnl5: failed waiting for nodepool="np-4smnl5" for cluster "cluster-candidate-4-23-4smnl5" in resourcegroup="rg-candidate-4-23-bqlkqphqlrhv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"name": "d9c906a9-62a2-4fec-8262-456dd2b5e51a",
"status": "Failed",
"startTime": "2026-11-13T04:02:01.972677601Z",
"endTime": "2026-11-13T04:21:05.56135744Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-4smnl5-nm98c-k5msg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-4smnl5-nm98c-k5msg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-bqlkqphqlrhv-candidate-4smnl5--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-4smnl5-nm98c-k5msg to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d9c906a9-62a2-4fec-8262-456dd2b5e51a
correlation ID: fc7ff880-2948-4807-a4fc-dcd90799df98
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpe...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 2 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e6d92904060>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: f5c2b31d-3614-44bc-8d24-b7b0bc55b26b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2e6d92904060>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: f5c2b31d-3614-44bc-8d24-b7b0bc55b26b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2d186fe117a0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: 43414ce7-64c3-4688-88ca-39da1413fc90
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x2d186fe117a0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOperatorIdentityRoleSets
correlation ID: 43414ce7-64c3-4688-88ca-39da1413fc90
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/h...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 2 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x1eb25ebe180>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 161d28ed-0b64-4007-9411-b5625c5cd6bb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:43]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x1eb25ebe180>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 161d28ed-0b64-4007-9411-b5625c5cd6bb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x391f95172ea0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 363bf3d3-c949-4f22-90ee-7ad33b4e32aa
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/operator_identity_rolesets.go:106]: failed to list HCP Operator Identity Role Sets
Unexpected error:
<*exported.ResponseError | 0x391f95172ea0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOperatorIdentityRoleSets'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOperatorIdentityRoleSets
correlation ID: 363bf3d3-c949-4f22-90ee-7ad33b4e32aa
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: AzureMachine status.initialization.provisioned is false; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.46
Unexpected error:
<*fmt.wrapError | 0x298ff8376680>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-hf5grb" in resourcegroup="rg-np-skip-minor-hf5grb-482lzd4v57gn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05",
"name": "a5613375-9443-4d68-9e17-0ff7a2c93d05",
"status": "Failed",
"startTime": "2026-11-13T07:18:53.583036999Z",
"endTime": "2026-11-13T07:38:01.651580211Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-nps-420-422-5r77b-9fj4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-nps-420-422-5r77b-zmkv2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-nps-420-422-5r77b-9fj4g: NotReady: AzureMachine status.initialization.provisioned is false\nMachine cluster-nps-420-422-5r77b-zmkv2: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
correlation ID: 9bc3ecfd-be4c-41ac-bac1-303b965f5bd5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.46
Unexpected error:
<*fmt.wrapError | 0x298ff8376680>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-hf5grb" in resourcegroup="rg-np-skip-minor-hf5grb-482lzd4v57gn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05",
"name": "a5613375-9443-4d68-9e17-0ff7a2c93d05",
"status": "Failed",
"startTime": "2026-11-13T07:18:53.583036999Z",
"endTime": "2026-11-13T07:38:01.651580211Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-nps-420-422-5r77b-9fj4g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cluster-nps-420-422-5r77b-zmkv2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-nps-420-422-5r77b-9fj4g: NotReady: AzureMachine status.initialization.provisioned is false\nMachine cluster-nps-420-422-5r77b-zmkv2: NotReady: AzureMachine status.initialization.provisioned is false\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5613375-9443-4d68-9e17-0ff7a2c93d05
correlation ID: 9bc3ecfd-be4c-41ac-bac1-303b965f5bd5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provi...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x164258d863c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xs6vdbh9ddbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01",
"name": "7558c7b9-327a-43c2-8293-9ac1255cbb01",
"status": "Failed",
"startTime": "2026-11-13T04:30:51.785572468Z",
"endTime": "2026-11-13T04:50:00.485199208Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
correlation ID: e0fc903a-bf7b-4e5b-a7e3-14066f61f36f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x164258d863c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xs6vdbh9ddbm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01",
"name": "7558c7b9-327a-43c2-8293-9ac1255cbb01",
"status": "Failed",
"startTime": "2026-11-13T04:30:51.785572468Z",
"endTime": "2026-11-13T04:50:00.485199208Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7558c7b9-327a-43c2-8293-9ac1255cbb01
correlation ID: e0fc903a-bf7b-4e5b-a7e3-14066f61f36f
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpe...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0x39262adcd200>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 90e0405d-cf96-4f5e-8535-4b7476f99e20
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0x39262adcd200>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'eastasia' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastasia/hcpOpenShiftVersions
correlation ID: 90e0405d-cf96-4f5e-8535-4b7476f99e20
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/h...full failure pattern: ERROR CODE: NoRegisteredProviderFound; detail message No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xfabd067a0f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
correlation ID: 6767984a-deb2-4f60-8891-913076f601ac
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
<*exported.ResponseError | 0xfabd067a0f0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: NoRegisteredProviderFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NoRegisteredProviderFound",
"message": "No registered resource provider found for location 'swedencentral' and API version '2024-06-10-preview' for type 'locations/hcpOpenShiftVersions'. The supported api-versions are '2026-10-01, 2026-09-01-preview, 2026-06-30-preview, 2025-12-23-preview, 2024-06-10-preview'. The supported locations are 'uksouth, switzerlandnorth, canadacentral, australiaeast, westeurope, centralindia, brazilsouth, eastus2, centraluseuap, eastus2euap'."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/swedencentral/hcpOpenShiftVersions
correlation ID: 6767984a-deb2-4f60-8891-913076f601ac
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x691f5c08960>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7ndhwr7f8prm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761",
"name": "3852afe7-3cef-47a2-b8aa-4d84a9736761",
"status": "Failed",
"startTime": "2026-11-13T06:16:07.058086055Z",
"endTime": "2026-11-13T06:35:15.725926802Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-g2fdm-7l2bn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-cilium-np-g2fdm-7l2bn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
correlation ID: f6bc5e83-47cf-48b3-a570-66ef10fd1d63
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x691f5c08960>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-7ndhwr7f8prm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761",
"name": "3852afe7-3cef-47a2-b8aa-4d84a9736761",
"status": "Failed",
"startTime": "2026-11-13T06:16:07.058086055Z",
"endTime": "2026-11-13T06:35:15.725926802Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-cilium-np-g2fdm-7l2bn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-cilium-np-g2fdm-7l2bn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-7ndhwr7f8prm--managed/providers/Microsoft.Compute/virtualMachines/cluster-cilium-np-g2fdm-7l2bn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3852afe7-3cef-47a2-b8aa-4d84a9736761
correlation ID: f6bc5e83-47cf-48b3-a570-66ef10fd1d63
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:161]: failed to create node pool "np-1" for fully private cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x26ec0f50ace0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fully-private" in resourcegroup="fully-private-7bcdq64qvfpx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078",
"name": "b11c3892-1773-4bc2-900f-66570c5c0078",
"status": "Failed",
"startTime": "2026-11-11T01:15:23.863928224Z",
"endTime": "2026-11-11T01:34:29.732010151Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9h4sc-vdszr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9h4sc-vdszr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
correlation ID: e0a628bb-254f-4199-8b05-5ab8214ab941
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private.go:161]: failed to create node pool "np-1" for fully private cluster "fully-private"
Unexpected error:
<*fmt.wrapError | 0x26ec0f50ace0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fully-private" in resourcegroup="fully-private-7bcdq64qvfpx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078",
"name": "b11c3892-1773-4bc2-900f-66570c5c0078",
"status": "Failed",
"startTime": "2026-11-11T01:15:23.863928224Z",
"endTime": "2026-11-11T01:34:29.732010151Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-9h4sc-vdszr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-9h4sc-vdszr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fully-private-7bcdq64qvfpx--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-9h4sc-vdszr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b11c3892-1773-4bc2-900f-66570c5c0078
correlation ID: e0a628bb-254f-4199-8b05-5ab8214ab941
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33f63d810100>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-7xq4wmsrcmrp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"name": "b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"status": "Failed",
"startTime": "2026-11-11T01:23:45.973097536Z",
"endTime": "2026-11-11T01:42:49.787552422Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-8b65r-bz5nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-8b65r-bz5nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
correlation ID: 5b1f6b6f-4c84-444d-b026-e149bc59ef51
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33f63d810100>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-7xq4wmsrcmrp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"name": "b5a5bcb7-009e-4cd8-998e-a353d7b56d57",
"status": "Failed",
"startTime": "2026-11-11T01:23:45.973097536Z",
"endTime": "2026-11-11T01:42:49.787552422Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-8b65r-bz5nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-8b65r-bz5nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-bz5nj to exist\nMachine cluster-np-1-8b65r-lptbf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-7xq4wmsrcmrp--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-8b65r-lptbf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5a5bcb7-009e-4cd8-998e-a353d7b56d57
correlation ID: 5b1f6b6f-4c84-444d-b026-e149bc59ef51
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0xe89ee532e20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-t7ttl6fk8v4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02",
"name": "4691ec6a-977d-4467-bb77-e056805ada02",
"status": "Failed",
"startTime": "2026-11-11T01:27:04.62054198Z",
"endTime": "2026-11-11T01:46:09.929731411Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-dkgxm-8trx6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-dkgxm-8trx6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
correlation ID: ad20e79e-c229-41ea-a8e2-d72265e902c5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
<*fmt.wrapError | 0xe89ee532e20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-t7ttl6fk8v4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02",
"name": "4691ec6a-977d-4467-bb77-e056805ada02",
"status": "Failed",
"startTime": "2026-11-11T01:27:04.62054198Z",
"endTime": "2026-11-11T01:46:09.929731411Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-1-dkgxm-8trx6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-1-dkgxm-8trx6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-8trx6 to exist\nMachine cluster-np-1-dkgxm-gx8pv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-t7ttl6fk8v4f--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-dkgxm-gx8pv to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4691ec6a-977d-4467-bb77-e056805ada02
correlation ID: ad20e79e-c229-41ea-a8e2-d72265e902c5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:192]: failed to create node pool "np-1" for fully private cluster with KV "full-priv-kv"
Unexpected error:
<*fmt.wrapError | 0x399db18e1c20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "full-priv-kv" in resourcegroup="full-priv-kv-xts7895q2w9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5",
"name": "6ee03316-feff-4093-9246-bef8a3716fd5",
"status": "Failed",
"startTime": "2026-11-13T06:16:32.006714258Z",
"endTime": "2026-11-13T06:35:35.830707205Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-whczx-4dxn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-whczx-4dxn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
correlation ID: 003842c9-adb0-44c1-b432-98cc60f79fe8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_fully_private_with_kv.go:192]: failed to create node pool "np-1" for fully private cluster with KV "full-priv-kv"
Unexpected error:
<*fmt.wrapError | 0x399db18e1c20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "full-priv-kv" in resourcegroup="full-priv-kv-xts7895q2w9x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5",
"name": "6ee03316-feff-4093-9246-bef8a3716fd5",
"status": "Failed",
"startTime": "2026-11-13T06:16:32.006714258Z",
"endTime": "2026-11-13T06:35:35.830707205Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-np-1-whczx-4dxn8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-np-1-whczx-4dxn8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/full-priv-kv-xts7895q2w9x--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-1-whczx-4dxn8 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6ee03316-feff-4093-9246-bef8a3716fd5
correlation ID: 003842c9-adb0-44c1-b432-98cc60f79fe8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc"
Unexpected error:
<*fmt.wrapError | 0x4de899b99e0>:
failed to create NodePool np-qzz6xc: failed waiting for nodepool="np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc" in resourcegroup="rg-candidate-4-21-qzgwvl6kwrmt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"name": "f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"status": "Failed",
"startTime": "2026-11-13T06:42:31.271421653Z",
"endTime": "2026-11-13T07:01:35.883206031Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-qzz6xc-57t85-pxcdm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-qzz6xc-57t85-pxcdm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
correlation ID: 99f225d1-e26c-43e5-9f87-e0f4765d3804
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc"
Unexpected error:
<*fmt.wrapError | 0x4de899b99e0>:
failed to create NodePool np-qzz6xc: failed waiting for nodepool="np-qzz6xc" for cluster "cluster-candidate-4-21-qzz6xc" in resourcegroup="rg-candidate-4-21-qzgwvl6kwrmt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"name": "f6d950c7-ad78-474d-87dd-0b73abc9a89e",
"status": "Failed",
"startTime": "2026-11-13T06:42:31.271421653Z",
"endTime": "2026-11-13T07:01:35.883206031Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-np-qzz6xc-57t85-pxcdm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-np-qzz6xc-57t85-pxcdm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-pxcdm to exist\nMachine cluster-np-qzz6xc-57t85-xsfgx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qzgwvl6kwrmt-candidate-qzz6xc--managed/providers/Microsoft.Compute/virtualMachines/cluster-np-qzz6xc-57t85-xsfgx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f6d950c7-ad78-474d-87dd-0b73abc9a89e
correlation ID: 99f225d1-e26c-43e5-9f87-e0f4765d3804
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-<id> to exist Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.46
Unexpected error:
<*fmt.wrapError | 0xbf3c7f945e0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-2klx7r" in resourcegroup="rg-np-version-upgrade-2klx7r-qhhp8zd44mhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"name": "2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"status": "Failed",
"startTime": "2026-11-13T04:32:00.613107854Z",
"endTime": "2026-11-13T04:51:00.48916791Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
correlation ID: 628c311f-b207-4ece-9d6b-da4a010d2a2c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.46
Unexpected error:
<*fmt.wrapError | 0xbf3c7f945e0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-2klx7r" in resourcegroup="rg-np-version-upgrade-2klx7r-qhhp8zd44mhd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"name": "2310612b-d88d-4e8b-9207-5c4d19d2a7c2",
"status": "Failed",
"startTime": "2026-11-13T04:32:00.613107854Z",
"endTime": "2026-11-13T04:51:00.48916791Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-npupgrade-4-20-lghrt-bmk6t: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\nMachine cluster-npupgrade-4-20-lghrt-ghw6t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-2klx7r-qhhp8zd44mhd-np-upgrade-2k-dfc4a0e9/providers/Microsoft.Compute/virtualMachines/cluster-npupgrade-4-20-lghrt-ghw6t to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2310612b-d88d-4e8b-9207-5c4d19d2a7c2
correlation ID: 628c311f-b207-4ece-9d6b-da4a010d2a2c
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2971806857e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rplp5xfl4lpm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233",
"name": "c08cc337-7979-4e36-800f-25b7ff101233",
"status": "Failed",
"startTime": "2026-11-13T06:56:27.715876172Z",
"endTime": "2026-11-13T07:15:36.395648013Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine cluster-autoscale-az-45l8j-q7vtn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-autoscale-az-45l8j-q7vtn: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
correlation ID: 66603ef9-ab0b-4acc-9e94-e684aebd41a8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2971806857e0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-rplp5xfl4lpm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233",
"name": "c08cc337-7979-4e36-800f-25b7ff101233",
"status": "Failed",
"startTime": "2026-11-13T06:56:27.715876172Z",
"endTime": "2026-11-13T07:15:36.395648013Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine cluster-autoscale-az-45l8j-q7vtn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine cluster-autoscale-az-45l8j-q7vtn: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c08cc337-7979-4e36-800f-25b7ff101233
correlation ID: 66603ef9-ab0b-4acc-9e94-e684aebd41a8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 0 · Nov 13: 1 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xc4ddaff8bb8>:
VerifyNodesReady failed: there are not ready nodes: cluster-agg-np-a-dz7h5-wdm8j (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-11-13T06:34:17Z)
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v94sn66nsnfk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5",
"name": "427fbf99-b011-448a-8853-e0e336f802e5",
"status": "Failed",
"startTime": "2026-11-13T06:16:51.858643855Z",
"endTime": "2026-11-13T06:35:55.74861517Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-agg-np-a-dz7h5-wdm8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-agg-np-a-dz7h5-wdm8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
correlation ID: 3d6b880e-0708-43ee-96e6-2869a452519b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:405]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xc4ddaff8bb8>:
VerifyNodesReady failed: there are not ready nodes: cluster-agg-np-a-dz7h5-wdm8j (status=False, reason=KubeletNotReady, message=container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?, since=2026-11-13T06:34:17Z)
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v94sn66nsnfk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5",
"name": "427fbf99-b011-448a-8853-e0e336f802e5",
"status": "Failed",
"startTime": "2026-11-13T06:16:51.858643855Z",
"endTime": "2026-11-13T06:35:55.74861517Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cluster-agg-np-a-dz7h5-wdm8j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cluster-agg-np-a-dz7h5-wdm8j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v94sn66nsnfk--managed/providers/Microsoft.Compute/virtualMachines/cluster-agg-np-a-dz7h5-wdm8j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/427fbf99-b011-448a-8853-e0e336f802e5
correlation ID: 3d6b880e-0708-43ee-96e6-2869a452519b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed outfull failure pattern: creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 0 · Nov 12: 1 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x179a7ca89ca0>:
creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out
{
msg: "creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out",
err: <*errors.StatusError | 0x179a7c45ea00>{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "etcdserver: request timed out",
Reason: "",
Details: nil,
Code: 500,
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:296]: failed to install cert-manager manifests
Expected success, but got an error:
<*fmt.wrapError | 0x179a7ca89ca0>:
creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out
{
msg: "creating ValidatingWebhookConfiguration /cert-manager-webhook: etcdserver: request timed out",
err: <*errors.StatusError | 0x179a7c45ea00>{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "etcdserver: request timed out",
Reason: "",
Details: nil,
Code: 500,
},
},
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "<vm>" (output: "000")full failure pattern: run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "<vm>" (output: "000") Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.38%1 of 42 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsNov 10: 0 · Nov 11: 1 · Nov 12: 0 · Nov 13: 0 · Nov 14: 0 · Nov 15: 0 · Nov 16: 0 · Nov 17: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.666s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
<*errors.errorString | 0x1502950d1900>:
run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
{
s: "run command \"e2e-runcommand-1794361068429775103\" failed with exit code 28 on VM \"private-ingress-test-vm\" (output: \"000\")",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.666s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
<*errors.errorString | 0x1502950d1900>:
run command "e2e-runcommand-1794361068429775103" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
{
s: "run command \"e2e-runcommand-1794361068429775103\" failed with exit code 28 on VM \"private-ingress-test-vm\" (output: \"000\")",
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||